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Abstract 


The  SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria  (MOSAIC)  is  a  manage¬ 
ment  approach  for  establishing  and  maintaining  confidence  that  key  objectives  will  be  achieved 
successfully.  The  Mission  Assurance  Analysis  Protocol  (MAAP)  is  one  of  the  assessments  in¬ 
cluded  in  MOSAIC.  A  MAAP  assessment  provides  a  systematic,  in-depth  analysis  of  the  potential 
for  success  in  distributed,  complex,  and  uncertain  environments  and  can  be  applied  across  the  life 
cycle  and  throughout  the  supply  chain.  It  produces  a  broad,  yet  detailed,  view  of  a  distributed 
project  or  process  and  provides  a  foundation  for  collaborative ly  managing  the  success  potential  of 
a  project  or  process  over  time.  With  MAAP,  an  operational  model  reflecting  the  current  state  is 
first  developed.  The  model  is  then  analyzed  to  establish  the  probability  of  achieving  key  objec¬ 
tives  as  well  as  to  identify  any  relevant  risks  and  opportunities  that  can  have  an  impact  on  the 
ability  to  achieve  key  objectives.  The  purpose  of  this  document  is  to  preview  the  framework,  or 
core  set  of  activities  and  outputs,  that  defines  a  MAAP  assessment.  Because  MAAP  is  a  work  in 
progress,  future  documents  will  reflect,  as  appropriate,  any  changes  in  the  protocol  or  its  underly¬ 
ing  concepts. 
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1  Introduction 


Mission  Assurance 

Analysis  Protocol 
(MAAP) 

SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria 
(MOSAIC)  is  a  management  approach  for  establishing  and  maintain¬ 
ing  confidence  that  objectives  will  be  achieved  successfully.  It  com¬ 
prises  a  suite  of  risk-based  methods  for  assessing  and  managing  com¬ 
plex  projects  and  processes.  The  Mission  Assurance  Analysis  Protocol 
(MAAP)  is  one  of  the  assessments  included  in  MOSAIC.^ 

MAAP  is  a  systematic,  in-depth,  risk-based  assessment  for  evaluating 
current  conditions  and  determining  whether  a  project  or  process^  is  on 
track  for  success.  MAAP  analyzes  the  potential  for  success  in  distri¬ 
buted,  complex,  and  uncertain  environments  and  can  be  applied  across 
the  life  cycle  and  throughout  the  supply  chain.  It  produces  a  rich,  in- 
depth  view  of  the  current  conditions  and  circumstances  affecting  a 
project’s  or  process’  potential  to  succeed.  A  MAAP  assessment  is 
complex  and  can  be  a  time-consuming  endeavor. 

A  MAAP  assessment  considers  a  broad  range  of  factors,  but  also  in¬ 
cludes  detailed  analysis  of  these  factors  from  multiple  viewpoints, 
providing  managers  with  a  wealth  of  information  about  their  project  or 
process  and  its  chances  for  success.  MAAP  assessment  results  are  suf¬ 
ficiently  detailed  to  support  the  development  of  collaborative  im¬ 
provement  plans  with  little  or  no  additional  data  from  other  assess¬ 
ments  or  analyses. 

Purpose  of  this 

Document 

MAAP  is  a  work  in  progress.  A  document  highlighting  its  underlying 
concepts  was  published  in  2005  [Alberts  2005].  This  technical  note 
builds  on  that  document  by  providing  a  preview,  or  early  draft,  of 
MAAP.  Changes  may  be  made  based  on  additional  pilots.  This  tech¬ 
nical  note  presents  the  basic  approach,  or  framework,  for  conducting  a 
MAAP  assessment  by  specifying  the  core  set  of  activities  that  must  be 
performed  and  their  resulting  outputs.  However,  this  document  does 
not  provide  step-by-step  procedures  for  conducting  a  MAAP  assess¬ 
ment.  Training  and  additional  documentation  focusing  on  how  to  con¬ 
duct  a  MAAP  assessment  are  planned  for  future  release. 

See  the  Mission  Diagnostic  Protocol:  A  Risk-Based  Approach  to  Assessing  the  Potential  for  Success  for  details 
about  another  MOSAIC  assessment  protocol  [Alberts  2008], 

The  term  process  as  used  in  this  document  refers  to  both  operational  and  business  processes. 
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Intended  Audience 


Structure  of  this 
Document 


The  primary  audience  for  this  technical  note  is  people  who  have  expe¬ 
rience  assessing  and  managing  risk  in  development  and  operational 
settings.  This  includes  people  who  oversee  complex  projects  and 
processes.  People  who  have  experience  with  or  are  interested  in  the 
following  topics  might  also  find  this  document  useful: 

•  methods  for  assessing  and  managing  risk  and  opportunity 

•  general  proj  ect  or  program  management 

•  success-driven  management  of  proj  ects  or  processes 


The  remainder  of  technical  note  is  divided  into  the  following  parts: 

•  MOSAIC — ^presents  background  information  about  MOSAIC  and 
its  assessment  methods 

•  Mission  Assurance  Analysis  Protocol  (MAAP)  describes  the 
key  activities  for  conducting  a  MAAP  assessment 

•  Summary  and  Further  Work — ^presents  a  brief  synopsis  of  re¬ 
search  and  development  activities  related  to  MOSAIC  and  MAAP 

•  Appendix  A:  Risk  Management  Concepts — provides  a  basic 
overview  of  risk  management  concepts  and  philosophy 

•  Appendix  B:  Key  Drivers  of  Success  and  Failure  defines  the 
concept  of  success  and  failure  drivers  and  describes  how  they  can 
be  used  in  a  MAAP  assessment 

•  Appendix  C:  Protocol  Structure  and  Nomenclature  describes 
the  standard  structure  and  naming  conventions  for  the  MAAP  data 
flows 


2  I  CMU/SEI-2008-TN-011 


2  MOSAIC 


Introduction 


A  New  Approach  for 
Today’s  Problem 
Space 


Focus  ON  Projects 
AND  Processes 


This  section  provides  background  information  about  the  body  of  re¬ 
search  underlying  MOSAIC  and  MAAP.  It  also  explains  key  concepts 
and  terminology  needed  to  understand  MAAP.  Specifically,  this  sec¬ 
tion  examines  the  following: 

•  basic  structure  of  MOSAIC  assessment  methods 

•  focus  on  managing  key  objectives 

•  success-oriented  philosophy  of  MOSAIC 

•  outcome  analysis 

•  uncertainty  analysis 

•  event  analysis 


Today’s  business,  project,  and  operational  environments  are  becoming 
increasingly  complex.  People  often  struggle  to  make  sense  of  this 
complexity,  which  places  many  critical  projects  and  processes  at  risk 
of  failing.  MOSAIC  is  a  management  approach  that  establishes  and 
maintains  confidence  that  objectives  will  be  achieved  successfully.  It 
comprises  a  suite  of  risk-based  methods  for  assessing  and  managing 
complex  projects  and  processes  [Alberts  2007]. 

MOSAIC  is  a  highly  flexible  approach  that  can  be  applied  across  the 
project  or  process  life  cycle  and  used  to  manage  projects  and  processes 
that  cross  organizational  boundaries.  It  is  designed  to  help  people  ana¬ 
lyze  tradeoffs  and  make  better  decisions  in  situations  that  have  a  high 
degree  of  uncertainty.  MAAP  is  one  of  the  assessments  included  in 
MOSAIC. 


To  date,  MOSAIC  research  and  development  activities  have  primarily 
focused  on  assessing  the  success  potential  of  projects  and  processes. 
As  a  result,  this  document  examines  how  MAAP  is  used  in  the  context 
of  projects  and  processes.  As  MAAP  is  used  in  other  contexts  (e.g.,  to 
assess  technology),  additional  guidance  will  be  provided. 
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Projects 


Processes 


Outcome 

Management 


In  MOSAIC,  a  project  is  defined  as  a  set  of  activities  that  produces  a 
unique  product  for  a  customer  or  delivers  a  service  that  is  tailored  for  a 
customer’s  needs.  A  project  is  often  executed  only  once.  For  example, 
when  an  organization  develops  a  software-intensive  system  for  a  spe¬ 
cific  customer,  its  management  charters  a  project  to  develop  that  sys¬ 
tem.  The  project  begins  with  the  initial  concept  for  the  system  and 
ends  when  the  system  is  satisfactorily  delivered  to  the  customer. 
Projects  can  range  from  small  software  development  projects  with  5  or 
10  people  to  a  large  U.  S.  Department  of  Defense  (DoD)  systems  de¬ 
velopment  program  that  includes  multiple  government  and  contractor 
organizations. 


In  contrast  to  a  project,  a  process  is  a  set  of  activities  that  is  typically 
executed  more  than  once.  Two  types  of  processes  are  considered  in 
MOSAIC:  business  and  operational  processes. 

In  this  document,  a  process  that  provides  a  core  business  function  is 
called  a  business  process.  For  a  healthcare  organization,  the  patient- 
care  workflow  is  considered  to  be  a  core  business  process  because  it 
directly  supports  the  mission  of  the  organization  (i.e.,  to  provide 
healthcare  services  to  patients). 

An  operational  process  indirectly  supports  the  mission  of  the  organi¬ 
zation.  It  is  not  part  of  the  organization’s  revenue-producing  processes. 
An  information  technology  (IT)  process  for  configuring  and  maintain¬ 
ing  an  organization’s  computing  infrastructure  is  an  example  of  an 
operational  process.  The  term  process  as  used  in  this  document  refers 
to  both  operational  and  business  processes. 


MOSAIC  methods  help  decision  makers  establish  and  maintain  a  rea¬ 
sonable  degree  of  confidence  that  projects  and  processes  will  success¬ 
fully  achieve  their  defined  objectives.  The  overarching  goal  of  this  ap¬ 
proach  is  to  ensure  that  the  eventual  outcome,  or  result,  satisfactorily 
achieves  the  objectives  being  pursued.  The  focus  on  managing  out¬ 
comes  enables  decision  makers  to  balance  potential  gain  being  pursued 
(i.e.,  opportunity)  against  the  potential  losses  that  can  occur  (i.e.,  risk) 
and  to  define  a  path  toward  achieving  success. 
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Assessment 
Protocols, 
Activities,  and 
Techniques 


Protocol  Flexibility 


Supporting 

Artifacts 


Each  MOSAIC  assessment  and  management  method  is  based  on  a 
specific  protocol.  As  used  in  this  context,  a  protocol  is  the  basic  ap¬ 
proach,  or  framework,  for  conducting  an  assessment  or  management 
method.  It  defines  the  sequence  of  activities  that  must  be  performed 
but  does  not  indicate  how  to  perform  those  activities.  You  can  think  of 
a  protocol  and  its  associated  activities  as  providing  the  basic  require¬ 
ments  for  conducting  an  assessment. 

A  technique  is  a  specific  practice  that  can  be  used  when  performing  a 
protocol  activity.  For  example,  consider  the  following  protocol  activi¬ 
ty:  Gather  data  from  people.  Many  interviewing  and  surveying  tech¬ 
niques  can  be  used  to  gather  data  from  people  who  are  knowledgeable 
about  a  subject.  The  objective  is  to  select  the  technique  that  is  most 
appropriate  for  your  circumstances.  In  some  cases,  an  interview  might 
be  the  best  choice,  while  in  other  instances  a  survey  that  people  com¬ 
plete  anonymously  would  be  more  appropriate.  Either  way,  you  get  the 
needed  information;  you  just  use  different  means  to  collect  it. 


While  you  can  use  a  single  technique  to  achieve  the  goals  of  a  given 
protocol  activity,  you  might  decide  to  combine  several  techniques  to 
meet  the  goals.  In  this  regard,  MOSAIC  offers  considerable  flexibility 
in  tailoring  an  assessment  to  a  particular  environment  or  set  of  cir¬ 
cumstances. 


When  you  conduct  any  technique,  you  will  likely  use  one  or  more  sup¬ 
porting  artifacts  to  gather,  analyze,  or  record  data.  Worksheets,  tem¬ 
plates,  and  tools  are  all  examples  of  supporting  artifacts.  Suppose  for 
the  protocol  activity  Gather  data  from  people  you  decide  to  conduct  an 
interview  with  a  set  of  carefully  chosen  participanfs.  During  fhe  inter¬ 
view  session,  you  frame  the  discussion  around  a  set  of  key  questions. 
That  list  of  questions,  which  is  essential  for  conducting  an  efficient  and 
effective  interview,  is  an  example  of  a  supporting  artifact. 
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Assessment  Protocols  (and  their  associated  activities),  techniques,  and  supporting 

Methods  artifacts  form  the  basis  for  assessment  methods  in  MOSAIC.  Figure  I 

shows  how  a  method  is  created  by  linking  techniques  and  supporting 
artifacts  with  a  protocol’s  activities.  The  collective  set  of  techniques 
and  artifacts  used  to  conduct  the  protocol  (represented  by  the  shaded 
boxes)  constitutes  a  method"^  for  that  protocol. 


Protocol 


Protocol  Activities 


Techniques 


Protocoi  A 


Activity  A.  1  Activity  A.2 


Supporting  Artifacts 


Figure  1:  A  Method  Consistent  with  Protocoi  A 


- - ► 

Activity  A.3 


Multiple  Methods  With  MOSAIC,  multiple  methods  can  be  consistent  with  a  given  pro- 

CONSISTENT  WITH  A  tocol,  as  illustrated  in  Figure  2.  A  common  protocol  forms  the  basis  for 

Protocol  the  methods  illustrated  in  Figure  1  and  Figure  2.  However,  the  two 

methods  incorporate  different  techniques  and  artifacts.  The  two  me¬ 
thods  accomplish  the  same  objectives  as  defined  by  the  common  pro¬ 
tocol  they  follow,  but  each  incorporates  a  unique  combination  of  tech¬ 
niques  and  artifacts. 


Protocol 


Protocol  A 

- - - ► 


Protocol  Activities  Activity  A.  1 


Activity  A.2  Activity  A.3 


Techniques 


Supporting  Artifacts 


Figure  2:  A  Second  Method  Consistent  with  Protocoi  A 


For  example,  the  Incident  Management  Mission  Diagnostic  [Dorofee  2008]  is  a  method  associated  with  the 
Mission  Diagnostic  Protocol. 
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Broad  Applicability 
OF  MAAP 


The  protocol  defined  in  this  document,  MAAP,  can  be  applied  to  many 
different  domains  and  types  of  problems.  To  date,  MAAP  has  been 
applied  in  the  cyber-security  domain,  and  portions  of  it  have  been  ap¬ 
plied  in  the  software-development  domain.  In  general,  the  flexible  de¬ 
sign  of  MOSAIC  assessment  and  management  methods  allows  them  to 
be  applied  in  a  variety  of  domains  and  environments,  across  the  life 
cycle  and  throughout  the  supply  chain.  The  main  focus  when  applying 
MAAP  in  any  domain  or  problem  space  is  to  assess  the  likelihood  that 
key  objectives  will  be  achieved  (1)  under  current  and  expected  condi¬ 
tions  as  well  as  (2)  when  subjected  to  the  occurrence  of  events. 
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2.1 


MISSIONS  AND  THEIR  OBJECTIVES 


What  is  a  Mission?  The  term  mission  has  different  meanings,  depending  on  the  context  in 

which  it  is  used.^  For  example,  mission  is  used  to  describe  any  of  the 

following: 

•  the  purpose  of  an  organization 

•  the  goals  of  a  specific  department  or  group  within  a  larger  organi¬ 
zation 

•  the  specific  result  being  pursued  when  executing  a  project  or 
process 

•  the  objectives  of  each  activity  in  a  work  process 

•  the  function  of  each  technology  (e.g.,  a  software-intensive  system) 
that  supports  a  project  or  process 


Network  of 
Missions 

•  core  business  missions  within  the  organization  will  be  achieved 

•  the  organization’s  overall  mission  will  be  accomplished 

The  network  of  missions  can  also  extend  across  multiple  organizations. 
For  example,  when  multiple  companies  collaborate  on  a  joint  venture, 
such  as  building  and  fielding  a  complex  software-intensive  system, 
they  pool  their  resources  toward  achieving  a  common  mission.  Each 
organization  must  balance  its  local  objectives  against  the  shared  set  of 
objectives  defined  by  the  common  mission. 


A  broad  network  of  missions  exists  within  all  organizations.  Success  at 
the  organizational  level  requires  ensuring  that  all  missions  within  the 
network  are  aligned.  Ensuring  alignment  among  an  organization’s 
missions  helps  establish  confidence  that  both 


MOSAIC  Definition 
OF  Mission 


The  mission  of  a  project  or  process  typically  comprises  three  distinct 
types  of  objectives:  (1)  product  or  service,  (2)  cost,  and  (3)  schedule. 
These  three  objectives  define  the  tangible  and,  in  many  cases,  measur¬ 
able  outcomes  being  pursued. 


Within  the  context  of  projects  and  processes,  we  define  mission  as  the 
set  of  objectives,  or  desired  outcome,  of  a  project  or  process  within 
one  organization  or  spanning  multiple  organizations.  Put  another  way, 
the  mission  defines  what  success  looks  like  for  a  project  or  process. 


We  assert  that  mission  is  a  recursive  concept. 
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Product  and 
Service  Objectives 


Cost  and  Schedule 
Objectives 


Picture  of  Success 


Product  objectives  define  the  nature  of  the  items  produced.  These  ob¬ 
jectives  are  often  referred  to  as  technical  objectives  in  the  software 
development  domain.  For  example,  if  you  are  developing  a  software¬ 
intensive  system,  the  product  (i.e.,  technical)  objectives  define  the 
functional  and  performance  characteristics  of  the  system  as  well  as 
other  desired  attributes,  like  safety  or  security.  Product  objectives  thus 
define  the  parameters  of  success  for  the  products  you  build. 

Service  objectives  define  the  nature  of  the  services  provided  to  the  re¬ 
cipients  of  those  services  (i.e.,  customers).  If  the  service  you  are  pro¬ 
viding  is  help-desk  support,  the  service  objectives  will  define  the  quali¬ 
ty  of  help-desk  support  provided  to  constituents  (such  as  the  required 
response  time  based  on  the  priority  of  the  request).  Service  objectives 
define  the  parameters  of  success  for  the  services  you  provide  to  cus¬ 
tomers. 


In  some  instances,  a  mission  is  defined  solely  by  its  product  or  service 
objectives.  Flowever,  in  most  cases,  constraints  are  also  considered  in 
relation  to  product  or  service  objectives.  Managers  generally  do  not 
have  unlimited  funds  at  their  disposal,  nor  do  they  have  an  infinite 
amount  of  time  in  which  to  complete  work  tasks.  As  a  result,  cost  and 
schedule  objectives  must  be  considered  alongside  product  or  service 
objectives  (and  in  many  cases  are  the  key  drivers  of  management’s 
decisions,  especially  as  time  goes  by  and  costs  accrue). 


Product  or  service,  cost,  and  schedule  objectives,  when  viewed  togeth¬ 
er,  typically  define  the  basic  mission  of  a  project  or  process.  They  spe¬ 
cify  what  will  be  accomplished,  the  anticipated  costs  to  complete  all 
activities,  and  the  time  frame  in  which  work  will  be  completed.  When 
appropriate,  these  objectives  can  be  supplemented  with  other  objec¬ 
tives  (such  as  business  or  financial  objectives)  to  produce  a  complete 
picture  of  success. 

The  mission,  or  picture  of  success,  defines  the  desired  outcome  for  a 
project  or  process.  Once  the  desired  outcome  is  established,  manage¬ 
ment  activities  must  be  geared  toward  ensuring  that  results  satisfy  that 
outcome.  Risk  management  is  an  essential  part  of  achieving  that  suc¬ 
cess.  (Appendix  A:  Risk  Management  Concepts  of  this  document  high¬ 
lights  the  foundational  concepts  of  risk  management  as  used  in 
MOSAIC.) 
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An  Incomplete 
Picture  Using 
Traditional  Risk 
Management 


Organizations  typically  manage  several  types  of  risk  using  traditional 
approaches,  including  project  risk,  security  risk,  technology  risk,  and 
operational  risk.  Each  type  of  risk  is  differentiated  by  the  unique 
sources,  or  causes,  that  produce  it.  Normally,  responsibility  for  manag¬ 
ing  different  types  of  risks  is  assigned  to  different  groups  within  an 
organization. 

Because  each  type  of  risk  is  normally  managed  in  isolation,  it  is  diffi¬ 
cult  to  establish  the  overall  success  potential  of  a  project  or  process 
using  traditional  risk-management  approaches.  Since  different  groups 
in  an  organization  have  responsibility  for  managing  different  types  of 
risk,  each  group  tends  to  locally  optimize  its  mitigation  efforts.  No  one 
is  responsible  for  consolidating  disparate  risk  data.  As  a  result,  the 
overall  chances  for  success  are  not  explicitly  determined.  In  contrast,  a 
MOSAIC  assessment  is  specifically  designed  to  establish  the  overall 
success  potential  of  a  project  or  process  by  analyzing  a  broad  range  of 
success  and  failure  drivers. 
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2.2  MANAGING  FOR  SUCCESS  USING  MOSAIC 


Introduction 


Assessment  Goals 


Success-Oriented 

Philosophy 


Potential  for 
Success 


Success  Criteria 


This  section  presents  a  few  of  the  key  concepts  underlying  the 
MOSAIC  management  approach.  These  ideas  provide  a  common 
foundation  for  all  MOSAIC  assessment  protocols.  Key  concepts  and 
features  unique  to  MAAP  are  highlighted  in  the  2.3. 


The  goal  of  all  MOSAIC  assessments,  including  MAAP  assessments, 
is  to  determine  the  success  potential  of  a  project  or  process.  This  focus 
on  managing  success  clearly  distinguishes  MOSAIC  from  traditional 
risk  management,  in  which  the  goal  is  to  avoid  failure.  A  key  aspect  of 
mosaic’s  success-oriented  approach  is  being  able  to  assess  a 
project’s  or  process’  overall  chances  of  succeeding. 


The  MOSAIC  management  approach  requires  establishing  and  main¬ 
taining  a  reasonable  degree  of  confidence  that  project  or  process  ob¬ 
jectives  will  be  achieved  successfully.  This  success-oriented  philoso¬ 
phy  requires  managers  to  focus  their  attention  on  managing  the  result, 
or  outcome,  of  a  project  or  process.  The  goal  is  to  ensure  that  the  even¬ 
tual  outcome  fulfills  the  objectives  being  pursued. 

Traditional  risk-management  approaches  generate  a  set  of  risks  for  a 
project  or  process.  Each  risk  in  the  set  is  a  cause-effect  pair  that  con¬ 
veys  the  potential  consequence  triggered  by  a  single  condition  or 
event.  In  contrast,  MOSAIC 

•  focuses  on  the  desired  outcome  (i.e.,  the  objectives  being  pursued) 

•  examines  the  range  of  conditions  and  potential  events  that  affect 
the  chances  of  achieving  the  desired  outcome 


The  potential  for  success  characterizes  the  likelihood,  or  probability, 
that  the  desired  outcome  will  be  achieved  or  exceeded.  It  can  be  ex¬ 
pressed  qualitatively  in  relation  to  a  set  of  success  criteria  or  quantita¬ 
tively,  depending  on  the  assessment  method  that  is  used. 


Success  criteria  define  a  set  of  qualitative  measures  used  to  character¬ 
ize  the  potential  for  success.  The  success  criteria  in  Figure  3  depict  a 
five-point  measurement  scale  used  to  interpret  each  applicable  measure 
for  the  potential  for  success. 
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Measure 

Description 

Excellent 

Conditions  are  extremely  favorable  for  a  successful  outcome  (-  >  95%  chance  of  success). 

High 

Conditions  are  favorable  for  a  successful  outcome  (-  75%  chance  of  success). 

Medium 

Conditions  are  mixed,  making  success  and  failure  equally  likely  (~  50%  chance  of  success). 

Low 

Conditions  are  not  favorable  for  a  successful  outcome  {-  25%  chance  of  success). 

Minimal 

Conditions  are  extremely  unfavorable  for  a  successful  outcome  {-  <  5%  chance  of  success). 

Figure  3:  Success  Criteria 


Success  Profile  and  A  basic  success  profile  depicts  the  current  potential  for  success  in  rela- 
SUCCESS  T HRESHOLD  tion  to  its  success  threshold  that  defines  the  desired,  or  target,  potential 

for  success.  The  basic  success  profile,  which  is  shown  in  Figure  4, 
separates  acceptable  values  of  the  potential  for  success  from  those  that 
are  considered  to  be  unacceptable;  it  provides  a  minimal  set  of  deci¬ 
sion-making  information.  Flere,  the  current  potential  for  success  is 
lower  than  the  success  threshold,  and  is  likely  unacceptable.  (An  ex¬ 
panded  success  profile  is  presented  in  2.3.) 


Excellent . ^ . 

High . 

Medium . 

Low . 

Minimal . 


-  Success  threshold 

(desired  potential  for  success) 


^  Success 
^  differential 


X 


Current  potential  for  success 


Figure  4:  Basic  Success  Profiie 


Success  As  depicted  in  Figure  4,  the  success  differential  is  a  measure  of  the 

Differential  current  potential  for  success  in  relation  to  the  desired  value  as  defined 

by  the  success  threshold.  The  success  differential  illustrates  the  degree 
of  improvement  that  will  be  required  to  position  a  project  or  process 
for  success.  In  this  example,  management  will  need  to  identify  actions 
to  improve  the  potential  for  success  from  Low  to  Fligh. 
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Managing  the 
Potential  for 
Success 


Mission  Success  and 
Mission  Assurance 


When  applying  the  MOSAIC  approach,  people 

•  assess  the  current  potential  for  success  in  relation  to  its  desired 
value  (i.e.,  its  success  threshold) 

•  take  planned  action,  when  appropriate,  to  bring  the  potential  for 
success  in  alignment  with  the  success  threshold 

MOSAIC  requires  people  to  track  the  potential  for  success  over  time 
and  take  appropriate  action  as  needed  to  ensure  that  the  potential  for 
success  is  kept  within  an  acceptable  tolerance. 


Mission  success  is  achieving  key  operational  objectives.  Mission  as¬ 
surance  is  having  justifiable  confidence  in  mission  success.  When 
viewed  within  the  context  of  a  project  or  a  process,  mission  assurance 
focuses  on  establishing  and  maintaining  an  appropriate  level  of  confi¬ 
dence  in  the  potential  for  achieving  the  project  or  process  objectives. 
MOSAIC  is  a  means  of  achieving  the  desired  level  of  assurance  for 
projects  or  processes. 
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2.3  MAAP  CONCEPTS 


Mission  Assurance  in  In  today’s  business  environment,  collaborations  and  partnerships 
Distributed  among  enterprises  are  commonplace.  Work  products  routinely  cross 

Environments  organizational  boundaries  in  these  distributed  ventures,  and  no  single 

person  or  group  has  complete  authority  over  the  end-to-end  work 
process.  In  addition,  each  group  participating  in  the  collaborative  ven¬ 
ture  is  supported  by  internal  and,  in  many  instances,  outsourced  ser¬ 
vice  providers.  MAAP  is  a  systematic  approach  for  assessing  the  po¬ 
tential  for  achieving  the  desired  level  of  assurance  of  success  in 
distributed  environments. 


Defining  Objectives  An  objective,  as  defined  in  MAAP,  is  the  result  that  you  intend  to 

IN  MAAP  achieve  at  a  future  point  in  time.  This  concept  is  illustrated  in  Figure  5, 

where  the  objective  is: 


By  the  end  of  the  initial  deployment  phase 
(6  months),  the  payroll  application  willful¬ 
ly  support  operations  at  Site  A. 


This  particular  objective  defines  an  operational  result,  or  outcome,  6 
months  in  the  future  for  a  payroll  application  that  is  being  developed. 
Likewise,  cost  and  product  objectives  (e.g.,  reliability  or  performance 
objectives)  should  be  defined  for  the  same  point  in  time.  Note  that  the 
schedule  information  is  embodied  in  the  other  objectives;  MAAP  does 
not  require  you  to  define  a  separate  objective  for  the  schedule. 


Desired  Outcome 

The  payroll  application  will  fully 
support  operations  at  Site  A. 

♦ 

• - m 

. I . \ . Time 

tcurrent  tobjective 

Today  The  end  of  the  initial  deployment  phase 

(6  months) 


Figure  5:  MAAP  Objective 
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Key  Objectives 


Operational  Model 


MAAP  is  a  systematic  approach  for  assessing  the  potential  for  achiev¬ 
ing  each  key  objective  in  a  distributed  environment.  A  key  objective  is 
defined  as  a  vital  outcome  for  a  project  or  program.  It  defines  a  core 
result  that  you  want  to  achieve  in  the  future  and  also  provides  a 
benchmark  against  which  success  will  be  judged.  A  collection  of  key 
objectives  defines  the  mission,  or  picture  of  success,  for  a  project  or 
process.  Because  MAAP  is  designed  to  assess  distributed  projects  and 
processes,  the  overall  set  of  key  objectives  normally  includes  the  key 
objectives  of 

•  each  individual  team  or  group 

•  the  end-to-end  proj  ect  or  process 

Once  the  key  objectives  are  established,  the  assessment  activities  and 
artifacts  are  geared  toward  assessing  the  likelihood  that  each  key  ob¬ 
jective  will  be  achieved. 


MAAP  requires  the  development  of  an  operational  model  (e.g.,  a 
swimlane  diagram  for  a  project  or  process)  to  establish  a  benchmark  of 
performance.  The  operational  model  documents  the  workflow  for  a 
project  or  process,  including  the 

•  key  objectives  that  must  be  achieved 

•  flow  of  work  products  throughout  the  project  or  process 

•  person  or  group  that  conducts  each  activity 

•  name  of  each  activity 

•  sequence  in  which  activities  occur 

•  trigger  that  initiates  an  activity 

The  operational  model  is  used  extensively  in  a  MAAP  assessment  to 
determine  the  success  potential  for  each  key  objective  throughout  the 
distributed  project  or  process.  Figure  6  shows  an  example  of  a  swim- 
lane  diagram,  which  is  one  type  of  diagramming  technique  used  to 
characterize  workflow  of  a  project  or  process.  A  swimlane  diagram  is 
useful  for  examining  the  flow  of  work  and  determining  where  work 
products  cross  organizational  boundaries.  Figure  6  depicts  Project  X, 
which  includes  activities  performed  by  three  different  organizations. 
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Figure  6:  Swimlane  Diagram 

Establishing  The  overarching  goal  of  MAAP  is  to  establish  a  reasonable  degree  of 

Confidence  confidence  that  the  key  objectives  of  a  distributed  project  or  process 

will  be  achieved.  Establishing  confidence  in  a  distributed  project  or 
process  requires  keeping  each  of  the  following  within  an  acceptable 
tolerance: 

1.  The  current  potential  for  success  for  each  key  objective 

2.  The  uncertainty  range  associated  with  the  current  potential  for 
success  for  each  key  objective 

3.  The  sensitivity  of  each  key  objective  to  potential  events 

Each  of  the  three  items  will  be  examined,  beginning  with  the  current 
potential  for  success. 
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Current  Potential 
FOR  Success 


•  shows  three  organizations  are  working  collaboratively  to  achieve  a 
common  set  of  objectives  (i.e.,  a  common  mission) 

•  depicts  the  network  of  missions  for  Project  X 

The  group  or  team  from  each  organization  must  balance  its  local  ob¬ 
jectives  against  the  shared  set  of  objectives  for  the  project.  MAAP 
helps  achieve  this  balance  by  determining  the  success  potential  of  each 
local  mission  as  well  as  the  success  potential  of  the  overall  mission. 


To  determine  the  likelihood  of  achieving  the  objectives  of  a  distributed 
project  or  process,  you  must  assess  the  likelihood  of  achieving  the  key 
objectives  of  (1)  each  individual  team  or  group  and  (2)  the  end-to-end 
project  or  process.  This  approach  is  illustrated  in  Figure  7,  which 


Success  potential  of  overall  Project  X  objectives 


Project  X 

Organization  A 

AM- 

[aT]—  >fA4k— 

L»(  A3''^ 

Success  potential  of  Organization  A 
project  objectives 

Organization  B 

— ►[  B1  I — 

B2  I - Jb3  I — 

- *{b4] - 

P>Ib5 

Success  potential  of  Organization  B 
project  objectives 

Organization  C 

Success  potential  of  Organization  C 
project  objectives 

-►|C1'] - »]'C2  - ►jc3  Y' 

Figure  7:  Success  Potential  throughout  a  Network  of  Missions 


Basic  Success 
Profile  for  Each 
Key  Objective 


As  shown  in  Figure  8,  a  basic  success  profile  that  shows  the  current 
potential  for  success  in  relation  to  its  success  threshold  is  developed 
for  each  key  objective  in  a  distributed  project  or  process.  This  forms 
the  basis  for  the  expanded  success  profile  generated  by  a  MAAP  as¬ 
sessment.  An  expanded  success  profile  also  includes  the  uncertainty 
range  and  event  sensitivity  for  each  key  objective. 
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Excellent  • 
High  ■ 
Medium  • 

Low  • 

Minimal  • 


Objective  1 


-  -  Success  threshold 

(desired  potential  for  success) 


I  X  I  Current  potential  for  success 


Excellent . 

High . 

Medium . 

Low . 

Minimal . 


Objective  2 


I  X  I  Current  potential  for  success 


-  Success  threshold 
{desired  potential  for  success) 


Figure  8:  Basic  Success  Profiie  for  Each  Key  Objective 


Uncertainty  Uncertainty  is  defined  as  having  doubt  or  being  unsure  of  something. 

When  conducting  a  MAAP  assessment,  you  collect  data  from  (1) 
people  who  are  knowledgeable  about  a  subject  or  problem  area  and  (2) 
generate  information  from  documentation  related  to  that  subject  or 
problem  area.  In  some  cases,  you  might  also  observe  people  as  they 
perform  their  day-to-day  work  tasks.  Invariably,  the  information  you 
collect  will  be  incomplete  to  some  degree.  As  you  analyze  informa¬ 
tion,  one  or  more  of  the  following  conditions  will  likely  be  true: 

•  Certain  information  is  not  available  or  is  unknown. 

•  You  do  not  trust  certain  information  based  on  its  source. 

•  Some  information  is  based  on  people’s  assumptions  or  opinions, 
which  might  prove  to  be  contradictory  or  incorrect. 

The  resulting  uncertainty  must  be  reflected  in  the  results  of  the  as¬ 
sessment. 


Uncertainty  Range  In  MAAP,  when  you  estimate  the  current  potential  for  success  for  a 

key  objective,  you  also  perform  an  uncertainty  analysis.  Because  of 
uncertainty,  the  actual  potential  for  success  might  deviate  from  the 
value  you  determined.  The  goal  of  the  uncertainty  analysis  is  to  deter¬ 
mine  the  best-  and  worst-case  scenarios  for  the  current  potential  for 
success  based  on  the  degree  of  uncertainty  inherent  in  the  distributed 
project  or  process.  The  best-  and  worst-case  scenarios  define  the  uncer¬ 
tainty  range  for  a  key  objective’s  current  potential  for  success.  The 
uncertainty  range  defines  fhe  highesf  and  lowesf  values  of  the  potential 
for  success  for  a  key  objective.  Figure  9  shows  a  success  profile  for  a 
key  objective  that  includes  an  uncertainty  range  from  Minimal  to  Me¬ 
dium.  For  this  example,  even  in  the  best-case  scenario,  the  potential  for 
success  is  below  the  success  threshold  of  High. 
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Figure  9:  Success  Profile  with  Uncertainty  Range 


Events  An  event  is  an  occurrence  that  changes  the  current  state  (i.e.,  changes 

the  status  quo)  for  a  project  or  process.  The  occurrence  of  an  event  can 
have  a  positive  or  negative  effect  on  the  outcome  depending  on  the 
specific  nature  of  the  event.  For  example,  an  increase  in  funding  would 
likely  be  perceived  as  a  positive  consequence  that  might  put  a  project 
in  better  position  for  success  (i.e.,  an  opportunity). 

On  the  other  hand,  a  decrease  in  funding  would  likely  be  perceived  as 
a  negative  consequence  that  might  adversely  affect  a  project’s  out¬ 
come  (i.e.,  a  risk).  The  current  potential  for  success  only  reflects  how 
present  conditions  are  influencing  the  outcome.  Analyzing  the  impact 
of  events  provides  a  complementary  view  by  examining  how  changing 
conditions  can  affect  the  potential  for  success. 


Example:  Analyzing  Effective  project  and  process  management  requires  anticipating  the 
Events  effects  of  potential  events  and  taking  action  to  ensure  that  each  key 

objective’s  potential  for  success  will  remain  within  an  acceptable  to¬ 
lerance  if  those  events  occur.  Figure  10  shows  how  the  occurrence  of 
an  event  can  affect  the  success  profile  for  a  key  objective.  In  the  fig¬ 
ure,  the  current  potential  for  success  is  low.  The  occurrence  of  a  par¬ 
ticular  event  lowers  the  potential  for  success  to  minimal.  The  event  in 
this  example  triggers  a  risk  for  the  project  or  process. 


SOFTWARE  ENGINEERING  INSTITUTE  |  19 


Objective  1 

Excellent . ; . 


High . i - Success  threshold 

i  {desired  potential  for  success) 

Medium . ; . 

Low . i .  I  X  I  Current  potential  for  success 

Minimal . ■ .  X  Potential  for  success  after  event 


Figure  1 0:  Success  Profile  with  Event  Sensitivity 

Importance  of  The  event  analysis  specified  by  MAAP  requires  you  to  examine  how 

Operational  Model  potential  events  will  likely  affect  a  distributed  project  or  process. 

IN  Event  Analysis  When  analyzing  events  as  part  of  MAAP,  you  use  the  operational 

model  to  determine  how  those  events  will  affect  the  current  value  of 
the  potential  for  success. 

Most  traditional  risk-analysis  approaches  rely  on  (1)  tacit  understand¬ 
ing  of  project  or  process  performance  and  (2)  guesswork  to  determine 
the  consequence  of  an  event.  Using  the  operational  model  during  event 
analysis  improves  your  ability  to  predict  how  a  given  event  might  af¬ 
fect  the  success  potential  of  a  distributed  project  or  process. 


Expanded  Success  The  expanded  success  profile  generated  by  a  MAAP  assessment  in- 
Profile  eludes  the  following  three  types  of  information  for  each  key  objective: 

1 .  The  current  potential  for  success 

2.  The  uncertainty  range  for  the  current  potential  for  success 

3.  Sensitivity  to  a  range  of  events 

Figure  1 1  provides  an  example  of  an  expanded  success  profile  for  a 
key  objective  that  includes  the  potential  for  success,  uncertainty  range, 
and  sensitivity  to  an  event. 
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Objective  1 


Excellent . ; - Success  threshold 

(desired  potential  for  success) 


High 

Medium 

Low 


^  Current  potential  for  success 
and  uncertainty  range 


Minimal 


Xevent  Potential  for  success  with  event 


Figure  1 1:  Expanded  Success  Profile  for  a  Key  Objective 


MAAP  Analysis  The  analysis  approach  embodied  in  MAAP  is  performed  in  two  parts: 

Approach 

1 .  The  current  potential  for  success  and  uncertainty  range  for  each 
key  objective  are  determined  by  analyzing  conditions  that  are  af¬ 
fecting  the  project  or  process. 

2.  The  potential  for  success  for  each  key  objective  is  determined  for 
a  range  of  events 
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3  Mission  Assurance  Anaiysis  Protocoi  (MAAP) 


Introduction 


Purpose 


Objectives 


This  section  describes  MAAP.  It  begins  with  an  overview  of  MAAP 
processes  and  activities.  Then,  details  for  key  activities  are  provided. 
Once  additional  MAAP  pilots  have  been  conducted,  this  “preview” 
version  of  the  protocol  will  be  re-issued  with  additional  details  and 
examples. 


A  MAAP  assessment  provides  a  systematic,  in-depth  evaluation  of  a 
distributed  project  or  process  to  identify  issues  or  circumstances  that 
can  affect  the  potential  for  success.  Key  results  of  a  MAAP  assessment 
include  an  operational  model,  customized  analysis  artifacts,  a  success 
profile  for  each  key  objective,  and  strategies  for  ensuring  that  each 
success  profile  will  be  acceptable  over  time.  Team  members  conduct¬ 
ing  a  MAAP  assessment  must  collectively  have  experience  and  exper¬ 
tise  in 

•  the  domain  area  being  assessed 

•  performing  MAAP  assessments 


The  main  objectives  of  a  MAAP  assessment  are  to 

•  assess  each  key  objective’s  potential  for  success  in  a  distributed 
project  or  process 

•  determine  whether  each  key  objective’s  current  potential  for  suc¬ 
cess  is  acceptable 

•  establish  the  uncertainty  range  for  each  key  objective’s  current 
potential  for  success 

•  analyze  the  sensitivity  of  each  key  objective  to  a  range  of  events 

•  identify  actions  to  maintain  or  improve  each  key  objective’s  cur¬ 
rent  potential  for  success 

•  provide  the  foundation  for  managing  each  key  objective’s  potential 
for  success  over  time 
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Assessment 

Benefits 


Assessment 

Limitations 


When  used  properly,  a  MAAP  assessment  provides  a  comprehensive 
diagnosis  of  the  issues  affecting  the  success  potential  of  a  project  or 
process  in  a  distributed  project  or  process. 

MAAP  is  designed  to  assess  a  distributed  project  or  process,  where 
multiple  groups  collectively  work  toward  a  common  set  of  objectives. 
The  protocol  requires  a  team  of  experts  to  examine  the  interactions, 
relationships,  and  dependencies  among  the  activities  in  a  distributed 
project  or  process.  It  also  requires  the  team  of  experts  to  analyze  the 
success  potential  of  (1)  each  individual  group  and  (2)  the  collection  of 
groups. 

A  MAAP  assessment  produces  a  comprehensive  set  of  findings,  which 
provides  a  solid,  accurate  foundation  for  creating  detailed  improvement 
strategies  and  plans. 


A  MAAP  assessment  must  be  conducted  by  an  analysis  team  that  col¬ 
lectively  has  considerable  experience  and  expertise  in  the  domain  area 
being  assessed  and  in  conducting  MAAP  assessments. 

MAAP  requires  considerable  depth  of  knowledge  about  risk  analysis 
and  management,  process  modeling,  and  statistics.  People  conducting  a 
MAAP  assessment  must  also  be  highly  skilled  analysts. 

Conducting  a  MAAP  assessment  is  a  time-  and  resource-intensive  en¬ 
deavor  for  the  analysis  team.  It  also  requires  a  considerable  time  com¬ 
mitment  from  the  people  who  are  working  on  the  project  or  process 
being  assessed.  These  people  provide  the  information  about  how  key 
activities  are  performed.  They  also  identify  key  strengths  and  weak¬ 
nesses  of  the  project  or  process.  Finally,  it  requires  access  to  documen¬ 
tation  and  information  with  respect  to  how  activities  are  performed  as 
well  as  activity  results. 
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Skills  Required  A  MAAP  assessment  is  normally  performed  by  a  small,  trained  team 

(referred  to  as  an  analysis  team)  with  the  following  skills:^ 

•  experience  with  MAAP  process,  techniques,  and  artifacts 

•  detailed  knowledge  of  the  domain  in  which  the  project  or  process 
is  executed 

•  ability  to  develop  accurate  models  of  process  or  project  activities 

•  knowledge  and  skill  with  risk  analysis  and  management  concepts, 
methods,  and  techniques 

•  basic  statistical  knowledge  and  experience 

•  analysis  expertise 

•  knowledge  of  process  improvement  and  management 

•  knowledge  and  skills  appropriate  to  applying  MAAP,  such  as 

-  interviewing  skills 

-  facilitation  skills 

-  note-taking  skills  (i.e.,  ability  to  quickly  record  data  that  are 
identified  by  participants) 

-  communication  skills 

MAAP  defines  a  relatively  complex  assessment.  It  should  not  be  un¬ 
dertaken  lightly  or  without  complete  understanding  of  the  required  re¬ 
sources  and  skills. 


These  skills  can  be  distributed  across  a  number  of  people  in  a  team.  Some  people  may  have  multiple  skills  and 
others  may  be  specialists.  What  is  important  is  that  the  team  performing  the  MAAP,  as  a  whole,  has  this  set  of 
skills. 
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3.1  PROTOCOL  STRUCTURE 


Phased  Assessment  The  goal  of  each  MOSAIC  assessment  protocol  is  to  specify  a  se- 
Approach  quence  of  activities  that  must  be  performed  when  conducting  that  as¬ 

sessment.  However,  an  assessment  is  performed  within  a  broader  con¬ 
text,  or  environment.  Therefore,  the  protocol  structure  used  within 
MOSAIC  also  specifies  preparation  and  post-assessment  activities. 
Figure  12  shows  the  three  phases  that  must  be  completed  when  con¬ 
ducting  MOSAIC  assessments. 


Phase  2 
Conduct  the 
Assessment 


\  Phase  3 

>  Complete  Post- 

i  Assessment  Activities 


Figure  12:  Protocol  Structure 


Protocol  The  focal  point  of  a  MOSAIC  assessment  protocol  is  a  dataflow  dia- 

Dataflows  gram.  For  each  assessment  protocol,  the  following  diagrams  are  docu¬ 

mented: 

•  a  high-level  dataflow  diagram  for  each  phase 

•  a  detailed  dataflow  diagram  for  Phase  2 

•  a  high-level  dataflow  diagram  for  each  Phase  2  activity 

Phase  2  is  described  in  more  detail  than  the  other  two  phases  because  it 
specifies  the  distinct  sequence  of  activities  that  uniquely  defines  the 
assessment  approach.  In  other  words,  the  unique  characteristics  of  the 
assessment  are  embodied  in  its  Phase  2  activity  dataflow.  The  prepara¬ 
tion  and  post-assessment  activities  of  Phases  1  and  3  are  common  to  all 
assessment  protocols  and  do  not  have  a  unique  sequence  of  activities 
associated  with  them.  Only  a  top-level  dataflow  is  presented  for  Phases 
1  and  3.  More  detailed  information  about  the  structure  of  MOSAIC 
assessment  protocols  is  presented  in  Appendix  C:  Protocol  Structure 
and  Nomenclature. 
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3.2  PREPARE  FOR  THE  ASSESSMENT  (PHASE  1) 


Introduction  Phase  1  of  a  MAAP  assessment,  Prepare  for  the  Assessment,  is  fo¬ 

cused  on  getting  ready  to  conduct  the  assessment.  This  includes  all  of 
the  planning  and  logistics  management  needed  to  make  the  assessment 
execution  flow  smoothly,  as  well  as  assuring  that  key  stakeholders 
provide  visible  support  for  the  assessment.  This  preparation  lays  the 
foundation  for  conducting  the  assessment  during  Phase  2. 


Objectives  Phase  1  answers  the  following  questions: 

•  Who  is  sponsoring  the  assessment? 

•  How  can  stakeholder  sponsorship  be  attained? 

•  What  is  the  scope  of  the  assessment? 

•  What  is  the  plan  for  conducting  the  assessment? 

•  How  will  the  assessment  team  gain  the  knowledge,  skills,  and  abil¬ 
ities  to  perform  the  assessment  (if  they  do  not  have  them  now)? 

•  What  facilities  and  equipment  are  needed  to  conduct  each  assess¬ 
ment  activity? 

•  What  procedures,  tools,  and  artifacts  are  needed  to  conduct  each 
assessment  activity? 

Dataflow  The  following  diagram  highlights  the  data  flow  for  this  protocol  phase. 


Constraint 

Cl  Assessment  constraints 


Input 

PRI1  Assessment  requirements 


Outputs 

PR01  Stakeholder  sponsorship 

PR02  Assessment  scope 

PROS  Assessment  plan 

PR04  Assessment  logistics 

PROS  Trained  personnel 

PROS  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Resources 

R1  Mission  Assurance  and  Analysis  Protocol  (MAAP) 
R2  MAAP  preparation  procedures 
R3  MAAP  preparation  artifacts  &  tools 
R4  MAAP  assessment  training  artifacts 
R5  Experienced  personnel 


Figure  13:  Dataflow  for  MAAP  Phase  1 
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Input 


The  following  input  is  required  by  the  activities  performed  during  this 
protocol  phase. 


Type 

Description 

PRI1  Assessment  require¬ 
ments 

The  goals  of  the  assessment,  needs  of  the  stakeholders,  and  a  basic  descrip¬ 
tion  of  the  project  or  process  being  analyzed 

Constraint  The  following  constraint  affects  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

Cl  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

Resources  The  following  resources  support  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

R1  Mission  Assurance 
Analysis  Protocol  (MAAPf 

The  basic  approach,  or  framework,  for  conducting  a  MAAP  assessment 

R2  MAAP  preparation  pro¬ 
cedures 

Documentation  that  describes  how  to  prepare  for  a  MAAP  assessment 

R3  MAAP  preparation  arti¬ 
facts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  prepare  for  a  MAAP 
assessment 

R4  Assessment  training 
artifacts 

Documentation  and  other  materials  used  to  train  people  how  to  conduct  a 

MAAP  assessment 

R5  Experienced  personnel 

People  who  are  experienced  in  all  phases  of  a  MAAP  assessment® 

Outputs  The  following  outputs  are  produced  by  the  activities  performed  during 

this  protocol  phase. 


Note  that  an  existing  method  consistent  with  the  protocol  will  include  all  of  the  procedures,  artifacts,  and  tools 
required  to  perform  the  assessment.  For  this  protocol,  it  is  assumed  a  method  is  created  as  part  of  preparation. 
If  a  method  already  exists  that  is  appropriate,  then  it  would  take  the  place  of  resources  R1 ,  R2,  and  R3. 

MAAP  defines  a  relatively  complex  assessment.  Team  members  conducting  a  MAAP  assessment  must  collec¬ 
tively  have  experience  and  expertise  in  (1)  the  domain  area  being  assessed  and  (2)  performing  MAAP  assess¬ 
ments.  MAAP  requires  considerable  depth  of  knowledge  about  risk  analysis  and  management,  process  model¬ 
ing,  and  statistics.  At  least  one  person  on  the  team  must  have  experience  and  expertise  in  applying  MAAP. 
Other  team  members  can  be  receive  training  in  MAAP  either  before  the  assessment  or  “just-in-time”  as  each 
protocol  activity  is  about  to  be  performed. 
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Type 

Description 

PR01  Stakeholder  spon¬ 
sorship 

Active  and  visible  support  of  the  assessment  by  key  stakeholders  and  decision 
makers 

PR02  Assessment  scope® 

The  boundaries  of  the  assessment,  including 

•  each  key  objective  for  the  project  or  process 

•  all  activities  needed  to  achieve  the  key  objectives 

•  the  people  who  have  ultimate  responsibility  for  completing  or  overseeing 
each  project  or  process  activity 

PROS  Assessment  plan 

The  approach  for  conducting  the  assessment,  including  key  activities,  re¬ 
sources,  schedule,  and  funding,  as  well  as  the  requirements  for  communicating 
results  to  key  stakeholders  after  the  assessment  is  complete 

PR04  Assessment  logistics 

The  facilities  and  equipment  needed  to  conduct  the  assessment  as  well  as 
communications  about  meeting  times  and  locations 

PROS  Trained  personnel 

The  people  who  are  tasked  with  performing  the  assessment  and  are  able  to 
conduct  it 

PROS  MAAP  assessment 
procedures 

Documentation  that  describes  how  to  conduct  assessment  activities 

PR07  MAAP  assessment 

artifacts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  perform  assessment 
activities 

Key  Activities  The  following  table  highlights  the  activities  performed  during  this  pro¬ 

tocol  phase. 


Activity 

Description 

Develop  stakeholder  spon¬ 
sorship 

Meet  with  key  stakeholders  and  decision  makers  to  foster  their  active  and  visi¬ 
ble  support  of  the  assessment 

Set  the  scope  of  the  as¬ 
sessment 

Determine  the  boundaries  of  the  assessment  based  on  requirements  and  con¬ 
straints  (schedule,  funding,  logistics,  contractual  restrictions) 

Develop  the  assessment 
plan 

Create  a  plan  for  conducting  the  assessment  based  on  its  scope  as  well  as 
requirements  and  constraints  (schedule,  funding,  etc.). 

Coordinate  logistics 

Reserve  rooms  for  meetings,  make  sure  that  any  required  equipment  (e.g., 
overhead  projectors,  flip  charts)  is  available,  and  inform  people  when  meetings 
will  be  held 

Train  personnel 

Ensure  that  people  who  will  perform  the  assessment  are  able  to  effectively 
conduct  all  assessment  activities 

The  scope  defines  which  activities  in  the  project  or  process  to  include  in  the  assessment  and  becomes  a  con¬ 
straint  in  Phase  2.  Some  aspects  of  a  project  or  process  might  be  excluded  from  an  assessment  due  to  contract 
limitations  or  on  the  basis  of  cost.  MAAP  is  designed  to  be  applied  to  distributed  projects  and  processes  and  to 
consider  both  local  and  collective  objectives.  Setting  the  scope  of  a  MAAP  assessment  also  includes  identifying 
the  various  geographic  locations,  entities,  organizations,  and  groups  that  will  be  included  in  the  assessment. 

Detailed  descriptions  of  Phase  1  activities  are  not  provided  in  this  document. 
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Activity 

Description 

Tailor  assessment  proce¬ 
dures,  criteria,  and  support¬ 
ing  artifacts” 

Adapt  all  MAAP  assessment  procedures,  criteria,  and  supporting  artifacts  (e.g., 
worksheets,  templates,  tools)  for  the  circumstances  and  contexts  in  which 
those  procedures  will  be  used 

MAAP  Tailoring  A  MAAP  assessment  must  be  tailored  for  the  context  in  which  it  is 

Considerations  applied.  The  table  below  highlights  some  areas  in  which  a  MAAP  as¬ 

sessment  is  commonly  tailored. 


Item 

Description 

Techniques 

The  specific  practices  used  to  perform  protocol  activities 

Selected  techniques  must  satisfactorily  achieve  the  key  outcomes  of  the  as¬ 
sessment  protocol  being  implemented. 

Procedures 

The  steps  followed  when  performing  a  technique 

Procedures  for  implementing  a  given  technique  must  be  consistent  with  the 
objectives  and  requirements  of  that  technique.  They  must  also  address  any 
constraints  and  unique  circumstances  encountered  (e.g.,  modifying  an  inter¬ 
view  technique  for  use  during  a  teleconference  rather  than  a  face-to-face  inter¬ 
view). 

Assessment  criteria 

A  set  of  measures  used  in  various  aspects  of  the  assessment  that  define  the 
permissible  range  of  values 

All  criteria  used  during  a  MAAP  assessment  must  reflect  the  requirements  and 
needs  of  key  decision  makers  and  stakeholders.  For  example,  a  wider  range  of 
values  for  success  criteria  could  be  used. 

Supporting  artifacts 

Worksheets,  templates,  and  tools  used  to  support  the  execution  of  a  given 
technique 

All  supporting  artifacts  must 

•  be  consistent  with  the  given  techniques  being  used 

•  support  the  key  outcomes  of  the  assessment  protocol  being  implemented 

•  support  the  overall  goals  of  the  assessment 

For  example,  artifacts  can  reflect  the  specific  language  and  terms  used  by  the 
project  or  can  be  automated  for  easier  analysis. 

11 


The  set  of  drivers  is  considered  to  be  an  assessment  artifact.  Tailoring  the  set  of  drivers  for  a  given  application 
of  MAAP  is  completed  during  Phase  1. 
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3.3  CONDUCT  THE  ASSESSMENT  (PHASE  2) 


Introduction 


Objectives 


Dataflow 


Inputs 

11  Peoples'  knowledge 

12  Documentation 


Figure  14: 


During  Phase  2,  the  core  assessment  activities  are  performed.  During 
this  phase,  a  success  profile  for  each  key  objective  is  established.  The 
success  profile  for  each  key  objective  includes 

•  The  potential  for  success  under  current  and  expected  conditions 

•  The  uncertainty  range  (i.e.,  best-  and  worst-case  scenarios)  for  the 
current  potential  for  success 

•  Sensitivity  to  a  range  of  events 

Decision-makers  then  determine  whether  the  success  profile  for  each 
key  objective  is  acceptable,  perform  a  tradeoff  analysis  when  appropri¬ 
ate,  and  identify  actions  for  maintaining  or  improving  each  success 
profile  over  time. 


This  protocol  phase  answers  the  following  questions: 

•  What  is  the  success  profile  (the  current  potential  for  success,  un¬ 
certainty  range,  and  sensitivity  to  events)  for  each  key  objective? 

•  Is  the  success  profile  for  each  key  objective  acceptable? 

•  How  can  the  success  profile  for  each  key  objective  be  maintained 
or  improved  over  time? 


The  following  diagram  highlights  the  dataflow  for  this  protocol  phase. 


Constraints 

Cl  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 


Phase  2 
Conduct  the 
assessment 


Resources 

PROS  Trained  personnel 

PROS  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Outputs 

01  Operational  model 
08  Next  steps 

For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 
05  Sensitivity  to  events 
06  Causal  analysis  for  event  sensitivity 
07  Success  profile 


Dataflow  for  MAAP  Phase  2 


30  I  CMU/SEI-2008-TN-011 


Inputs 


The  following  inputs  are  required  by  the  activities  performed  during 
this  protocol  phase. 


Type 

Description 

11  People’s  knowledge 

People’s  individual  and  collective  perspectives,  information,  and  opinions  about 
the  project  or  process  and  its  potential  for  success 

I2  Documentation 

Documentation  that  is  relevant  to  the  project  or  process.  Examples  include 
mission  statement,  policies,  procedures,  process  workflow,  work  products,  and 
quality  assurance  data. 

Constraints  The  following  constraints  affect  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

Cl  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

PR01  Stakeholder  spon¬ 
sorship 

Active  and  visible  support  of  the  assessment  by  key  stakeholders  and  decision 
makers 

PR02  Assessment  scope 

The  boundaries  of  the  assessment,  including 

•  each  key  objective  for  the  project  or  process 

•  all  activities  needed  to  achieve  the  key  objectives 

•  the  people  who  have  ultimate  responsibility  for  completing  or  overseeing 
each  project  or  process  activity 

PR03  Assessment  plan 

The  approach  for  conducting  the  assessment,  including  key  activities,  re¬ 
sources,  schedule,  and  funding,  as  well  as  the  requirements  for  communicating 
results  to  key  stakeholders  after  the  assessment  is  complete 

PR04  Assessment  logistics 

The  facilities  and  equipment  needed  to  conduct  the  assessment  as  well  as 
communications  about  meeting  times  and  locations 

Resources 


The  following  resources  support  execution  of  the  activities  performed 
during  this  protocol  phase. 


Constraints  affect  all  activities  performed  during  Phase  2,  while  resources  are  used  to  aid  the  completion  of  all 
activities  performed  during  Phase  2.  The  definitions  for  all  Phase  2  constraints  and  resources  are  provided  in 
this  section  only.  They  are  not  provided  in  the  sections  for  individual  Phase  2  activities. 
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Type 

Description 

PROS  Trained  personnel 

The  people  who  are  tasked  with  performing  the  assessment  and  are  able  to 
conduct  it 

PROS  MAAP  assessment 
procedures 

Documentation  that  describes  how  to  conduct  assessment  activities 

PR07  MAAP  assessment 

artifacts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  perform  assessment 
activities 

Outputs  The  following  outputs  are  produced  by  the  activities  performed  during 

this  protocol  phase. 


Type 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  of  each  team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

02  Current  potential  for 
success  (for  each  key  ob¬ 
jective) 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded.  The  current  potential  for  success  is  determined  for  each  key  ob¬ 
jective. 

03  Causal  analysis  of  cur¬ 
rent  state  (for  each  key 
objective) 

The  conditions  and  circumstances  that  are  driving  the  current  potential  for  suc¬ 
cess.  A  causal  analysis  of  the  current  state  is  developed  for  each  key  objective. 

04  Uncertainty  range  and 
rationale  (for  each  key 
objective) 

The  best-  and  worst-case  scenarios  for  the  current  potential  for  success  based 
on  the  degree  of  uncertainty  inherent  in  the  distributed  project  or  process  and 
the  justification  underlying  the  best-  and  worst-case  scenarios.  An  uncertainty 
range  is  determined  for  each  key  objective. 

05  Sensitivity  to  events  (for 
each  key  objective) 

The  potential  for  success  for  each  key  objective  given  the  occurrence  of  an 
event.  The  sensitivity  to  multiple  events  is  analyzed. 

06  Causal  analysis  for 
event  sensitivity  (for  each 
key  objective) 

The  conditions  and  circumstances  that  are  driving  the  potential  for  success  for 
each  key  objective  given  the  occurrence  of  an  event 

07  Success  profile  (for 
each  key  objective) 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  for  success,  or  success  threshold 

•  the  uncertainty  range  (i.e.,  best-  and  worst-case  scenarios)  for  the  current 
potential  for  success 

•  sensitivity  to  a  range  of  events 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold,  the  uncertainty  range,  and  the  sensitivity  to  events 

A  success  profile  is  developed  for  each  key  objective. 

08  Next  steps 

Actions  to  be  taken  after  the  assessment  is  complete 
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Key  Activities 


The  following  table  highlights  the  activities  performed  during  this  pro¬ 
tocol  phase.  The  remainder  of  this  section  provides  additional  details 
about  the  activities  featured  in  the  dataflow. 


Activity 

Description 

A1  Develop  operational 
model 

Create  a  detailed  operational  model  of  the  distributed  project  or  process  using 
data  gathered  from  people  who  play  a  role  in  executing  the  activities  and  do¬ 
cumentation  relevant  to  the  project  or  process  (policies,  procedures,  or  reports). 

A2  Identify  strengths  and 
weaknesses 

Determine  the  conditions  and  circumstances  that  are  affecting  the  execution  of 
the  distributed  project  or  process  (both  positively  and  negatively). 

A3  Perform  outcome  analy¬ 
sis 

For  each  key  objective,  determine  the  current  potential  for  success,  perform  a 
causal  analysis  to  determine  the  conditions  and  circumstances  that  are  driving 
the  current  potential  for  success,  and  establish  the  uncertainty  range. 

A4  Perform  event  analysis 

For  each  key  objective,  determine  the  potential  for  success  for  a  range  of 
events  and  perform  a  causal  analysis  to  determine  the  conditions  and  circums¬ 
tances  that  are  driving  the  potential  for  success  for  each  event. 

A5  Establish  success  pro¬ 
file 

Generate  a  success  profile  for  each  key  objective  by 

•  setting  the  success  threshold 

•  comparing  the  current  potential  for  success  to  the  success  threshold 

•  comparing  the  uncertainty  range  (i.e.,  best-  and  worst  case  scenarios  for 
the  current  potential  for  success)  to  the  success  threshold 

•  comparing  the  potential  for  success  for  each  event  to  the  success  thre¬ 
shold 

•  deciding  whether  or  not  the  current  success  profile  is  acceptable 

A6  Determine  next  steps 

Identify  actions  to  be  taken  after  the  assessment  is  complete  to  maintain  or 
improve  the  current  potential  for  success. 

Detailed  Dataflow  Figure  15:  Detailed  Dataflow  for  MAAP  Phase  2  provides  a  de¬ 

tailed  dataflow  for  MAAP  Phase  2. 
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C1  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PR03  Assessment  plan 
PR04  Assessment  logistics 


Phase  2  Conduct  the  assessment 


Al  ^ 
Develop 
operational 


01  Operational  model 


11  People’s  knowledge  _ 

12  Documentation 


For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 


01  Operational  model 
N1  Strengths  and  weaknesses 


A4 

Perform  event 
analysis 


For  each  key  objective: 

05  Sensitivity  to  events 

06  Causal  analysis  for  event  sensitivity 


01  Operational  model 
N1  Strengths  and  weaknesses 
For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 


A5 

Establish 
success  profile 


For  each  key  objective: 
07  Success  profile 


01  Operational  model 
N1  Strengths  and  weaknesses 
For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 
05  Sensitivity  to  events 
06  Causal  analysis  for  event  sensitivity 


A6 

Determine 
next  steps 


08  Next  steps 


PR05  Trained  personnel 
PR06  MAAP  assessment  procedures 
PR07  MAAP  assessment  artifacts  &  tools 


Figure  1 5:  Detailed  Dataflow  for  MAAP  Phase  2 
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3.3.1  Develop  Operational  Model  (Activity  A1) 


Introduction  This  protocol  activity  develops  an  operational  model  of  the  project  or 

process  being  assessed  using  information  from 

•  People  who  work  on  a  proj  ect  or  process 

•  Proj  ect  or  process  documentation 

•  (Optional)  Observing  people  as  they  perform  key  tasks  and  activi¬ 
ties 


Objectives  This  activity  answers  the  following  questions: 


•  What  are  the  key  objectives  for  the  project  or  process  and  selected 
teams  or  groups? 

•  What  activities  are  performed  in  the  project  or  process? 

•  What  is  the  sequence  of  activities? 

•  What  work  products  are  required  by  each  activity? 

•  What  work  products  are  produced  by  each  activity? 

•  What  additional  details  are  relevant  to  each  activity  (e.g.,  triggers, 
completion  criteria,  procedures,  training  required,  interface  de¬ 
scriptions)? 


3. 3. 1.1  Dataflow 


Constraints 

Cl  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 


inputs 

11  People's  knowledge 

12  Documentation 


A1 

Develop  Operational 
Model 


Output 

01  Operational  Model 


Resources 

PROS  Trained  personnel 

PROS  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Figure  1 6:  Inputs  and  Outputs  for  Activity  A 1 
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Inputs 

Description 

11  People’s  knowledge 

People’s  individual  and  collective  perspectives,  information,  and  opinions  about 
the  project  or  process  and  its  potential  for  success 

I2  Documentation 

Documentation  that  is  relevant  to  the  project  or  process.  Examples  include 
mission  statement,  policies,  procedures,  process  workflow,  work  products,  and 
quality  assurance  data. 

Output 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  the  key  objectives  of  each 
team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

3.3. 1.2  Techniques 

Developing  an 

Operational  Model 

Vlany  types  of  operational  models  can  be  used  to  represent  a  project  ot 
jrocess,  including  data  flows,  work  process  flows  [Sharpe  2001],  and 
communication  models.  The  type  of  model  you  decide  to  use  when  yo 
conduct  a  MAAP  assessment  will  depend  upon  the  objectives  of  the 
assessment,  the  kind  of  information  that  is  available,  and  the  target  of 
the  assessment  (e.g.,  a  project  or  process).  In  general,  when  you  are 
developing  an  operational  model,  you  can  use  information  generated 

Tom 

discussions  or  interviews  with  people  who  work  on  the  project  or 
process  being  assessed 

documentation  related  to  the  project  or  process  being  assessed 

observing  people  as  they  perform  project  or  process  tasks  and  ac¬ 
tivities  (optional) 

Developing  an  operational  model  is  an  interactive  and  iterative 
jrocess.  You  must  continually  verify  the  model  with  people  who  per¬ 
form  project  or  process  tasks  and  activities  to  ensure  that  the  model  is 
an  accurate  reflection  of  the  project  or  process  being  assessed. 

Refining  Key 

Objectives 

nformation  about  key  objectives  is  initially  collected  during  Phase  1: 
Prepare  for  the  Assessment.  During  Phase  2,  the  key  objectives  can  be 
refined,  if  needed.  The  goal  is  to  ensure  that  each  key  objective  is  mea 
ningful  and  represents  a  desired  outcome  for  the  project  or  process. 
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3.3.2  Identify  Mission  Strengths  and  Weaknesses  (Activity  A2) 


Introduction  In  order  to  analyze  the  potential  for  success,  you  must  identify  the 

conditions  and  circumstances  that  are  guiding  a  project  or  process  to¬ 
ward  success  (i.e.,  strengths)  and  those  that  are  guiding  a  project  or 
process  toward  failure  (i.e.,  weaknesses).  This  protocol  activity  identi¬ 
fies  strengths  and  weaknesses  using  information  from  (1)  people  who 
work  on  a  project  or  process,  (2)  project  or  process  documentation, 
and,  optionally,  (3)  observing  people  as  they  perform  key  tasks  and 
activities. 

Objectives  This  activity  answers  the  following  questions: 

•  What  conditions  and  circumstances  are  driving  the  project  or 
process  toward  a  successful  outcome? 

•  What  conditions  and  circumstances  are  driving  the  project  or 
process  toward  an  unsuccessful,  or  failed,  outcome? 


3. 3.2.1  Dataflow 


Constraints 

Cl  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 


Inputs 

11  People’s  knowledge 

12  Documentation 


A2 

Identify  mission 
strengths  and 
weaknesses 


Output 

N1  Strengths  and  weaknesses 


Resources 

PROS  Trained  personnel 

PROS  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Figure  1 7:  Inputs  and  Outputs  for  Activity  A2 


Inputs 

Description 

11  People’s  knowledge 

People’s  individual  and  collective  perspectives,  information,  and  opinions  about 
the  project  or  process  and  its  potential  for  success 

12  Documentation 

Documentation  that  is  relevant  to  the  project  or  process.  Examples  include 
mission  statement,  policies,  procedures,  process  workflow,  work  products,  and 
quality  assurance  data. 
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Output 

Description 

N1  Strengths  and  weak¬ 
nesses 

Conditions  and  circumstances  that  are  guiding  a  project  or  process  towards 
success  (i.e.,  strengths)  and  failure  (i.e.,  weaknesses).  Examples  of  strengths 
include  a  highly  trained  workforce  and  documented  policies  and  procedures. 
Examples  of  weaknesses  include  inefficient  workflow  layout  and  inadequate 
resource  allocation. 

3.3.2.2  Techniques 


T ECHNIQUES  The  techniques  employed  when  conducting  this  protocol  activity  de¬ 

pend  upon  the  nature  of  the  project  or  process  being  assessed,  the 
knowledge,  skills,  and  abilities  of  the  people  who  are  performing  the 
assessment,  as  well  as  organizational  practices,  culture,  and  con¬ 
straints.  The  objective  of  this  protocol  activity  is  to  identify  conditions 
and  circumstances  that  are  guiding  a  project  or  process  toward  success 
(i.e.,  strengths)  and  those  that  are  guiding  a  project  or  process  toward 
failure  (i.e.,  weaknesses).*^  The  following  classes  of  techniques  are 
normally  employed  when  conducting  this  protocol  activity: 

•  Data  collection  from  people 

•  Document  analysis 

•  Task  and  activity  observation  (when  appropriate) 

•  Preliminary  data  analysis 

Some  of  the  more  common  data  collection  techniques  include  work¬ 
shops,  interviews,  and  surveys.  Each  is  described  below: 

•  Workshop — facilitated  session  usually  focused  on  solving  one  or 
more  issues  or  problems.  A  workshop  tends  to  foster  a  collabora¬ 
tive  environment  between  the  facilitator  and  participants. 

•  Interview — a  facilitated  session  using  a  series  of  specific  questions 
asked  by  one  or  more  interviewers.  An  interview  tends  to  be  more 
formal  than  a  workshop  and  is  normally  focused  on  data  elicitation 
rather  than  problem  solving. 

•  Survey — a  list  of  written  questions  to  which  people  respond. 
People  responding  to  a  survey  have  little  interaction  with  those 
who  are  collecting  the  information,  making  surveys  a  rapid,  but 
impersonal,  means  of  collecting  data. 


Data  Collection 
FROM  People 


13 


Activities  A1  and  A2  can  be  performed  in  parallel  or  at  the  same  time.  For  example,  the  same  workshop  can  be 
used  to  gather  information  related  to  the  development  of  an  operational  model  as  well  as  information  about 
strengths  and  weaknesses. 
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Document  Analysis 

Document  analysis  involves  reviewing  documentation  relevant  to  the 
distributed  project  or  process  being  assessed.  When  you  review  a  given 
document,  you  normally  frame  the  analysis  around  an  explicit  set  of 
guidelines  or  questions.  The  guidelines  or  questions  you  use  must  be 
appropriate  for  generating  sufficient  data  about  the  specific  subject  or 
problem  area  being  investigated.  Alternatively,  you  can  use  implicit 
guidelines  during  document  analysis.  Here,  you  use  your  expertise  and 
experience  to  look  for  data  that  would  be  useful  to  the  assessment. 

Task  AND  Activity 

Observation 

In  some  instances,  you  might  decide  to  gather  data  by  observing  how 
people  perform  their  assigned  tasks  and  activities.  You  need  to  be  care¬ 
ful  when  watching  people  perform  tasks  and  activities  because  the 
mere  act  of  observing  them  generally  influences  their  actions  and  can 
skew  findings.  In  addition,  observing  people  without  notifying  them 
beforehand  can  lead  to  mistrust  on  the  part  of  those  who  have  been 
watched.  A  lack  of  tmst  can  adversely  affect  the  degree  to  which 
people  will  cooperate  with  data  gathering  activities.  However,  when 
used  judiciously,  targeted  observations  can  provide  useful  insight  into 
a  project  or  process,  especially  for  technical  tasks. 

Preliminary  Data 

Analysis 

A  considerable  amount  of  data  can  be  collected  from  people  and  do¬ 
cumentation  as  well  as  through  targeted  observations  of  key  tasks  and 
activities.  Often,  you  will  need  to  perform  a  preliminary  data  analysis 
to  convert  raw  data  into  a  set  of  strengths  and  weaknesses.  Subsequent 
analyses  can  be  conducted  more  efficiently  and  effectively  when  extra¬ 
neous  data  have  been  removed  from  the  data  set. 

Group  Decision 

Making 

When  performing  preliminary  data  analysis  in  a  group  setting,  you  can 
use  techniques  to  facilitate  decision-making  activities.  For  example, 
voting  techniques,  such  as  multi-voting,  can  help  a  group  sort  through 
differences  and  reach  consensus. 
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3.3.3  Perform  Outcome  Analysis  (Activity  A3) 

Introduction  This  protocol  activity  determines  the  current  potential  for  success  for 

each  key  objective  based  on  a  set  of  criteria,  called  success  criteria. 
Next,  an  analysis  is  performed  to  determine  which  conditions  and  cir¬ 
cumstances  are  driving  the  current  potential  for  success  for  each  key 
objective.  Finally,  the  uncertainty  range  for  each  key  objective  is  de¬ 
termined  by  estimating  the  best-  and  worst-case  values  of  the  potential 
for  success  and  providing  justification  for  those  values. 

Objectives  This  activity  answers  the  following  questions: 

•  What  is  the  current  potential  for  success  for  each  key  objective? 

•  What  conditions  and  circumstances  are  driving  the  current  poten¬ 
tial  for  success  for  each  key  objective? 

•  What  is  the  uncertainty  range  (i.e.,  the  best-  and  worst-case  values 
of  the  potential  for  success)  for  each  key  objective? 

•  What  unknowns  are  driving  the  uncertainty  range  for  each  key 
objective? 


3. 3. 3.1  Dataflow 


Constraints 

C1  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 


inputs 

01  Operational  model 
N1  Strengths  and  weaknesses 


A3 

Perform  outcome 
analysis 


Outputs 

For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 


Resources 

PROS  Trained  personnel 

PROS  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Figure  1 8:  Inputs  and  Outputs  for  Activity  A3 
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Inputs 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  the  key  objectives  of  each 
team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

N1  Strengths  and  weak¬ 
nesses 

Conditions  and  circumstances  that  are  guiding  a  project  or  process  towards 
success  (i.e.,  strengths)  and  failure  (i.e.,  weaknesses).  Examples  of  strengths 
include  a  highly  trained  workforce  and  documented  policies  and  procedures. 
Examples  of  weaknesses  include  inefficient  workflow  layout  and  inadequate 
resource  allocation. 

Outputs 

Description 

02  Current  potential  for 
success  (for  each  key  ob¬ 
jective) 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded.  The  current  potential  for  success  is  determined  for  each  key  ob¬ 
jective. 

03  Causal  analysis  of  cur¬ 
rent  state  (for  each  key 
objective) 

The  conditions  and  circumstances  that  are  driving  the  current  potential  for  suc¬ 
cess.  A  causal  analysis  of  the  current  state  is  developed  for  each  key  objective. 

04  Uncertainty  range  and 
rationale  (for  each  key 
objective) 

The  best-  and  worst-case  scenarios  for  the  current  potential  for  success  based 
on  the  degree  of  uncertainty  inherent  in  the  distributed  project  or  process  and 
the  justification  underlying  the  best-  and  worst-case  scenarios.  An  uncertainty 
range  is  determined  for  each  key  objective. 

3. 3. 3.2  Techniques 

T ECHNIQUES  The  techniques  employed  when  conducting  this  protocol  activity  de¬ 

pend  upon  the  knowledge,  skills,  and  abilities  of  the  people  who  are 
performing  the  assessment.  Determining  the  outcome  for  each  key  ob¬ 
jective  requires  techniques  for  analyzing  data  that  have  been  collected 
during  earlier  activities  in  relation  to  the  operational  model.  In  colla¬ 
borative  settings,  group  decision-making  techniques  can  also  be  effec¬ 
tive. 
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Preliminary  Data 
Analysis  Using  Key 
Drivers  of  Success 


Outcome  Analysis 


Causal  Analysis 


A  considerable  amount  of  data  can  be  collected  when  developing  an 
operational  model  for  a  project  or  process  and  when  identifying  key 
strengths  and  weaknesses  that  are  currently  affecting  its  execution.  A 
preliminary  data  analysis  based  on  a  small  set  of  outcome  drivers  can 
help  focus  subsequent  data  analysis  activities. 

A  driver  is  a  characteristic  of  a  project  or  process  that  is  essential  for 
achieving  its  objectives.  Each  individual  driver  has  a  strong  influence 
on  the  ultimate  outcome,  or  result.  The  cumulative  effects  of  all  drivers 
can  be  analyzed  to  determine  whether  a  project  or  process  has  suffi¬ 
cient  momentum  toward  its  objectives.  The  results  of  the  driver  analy¬ 
sis  can  be  used  as  an  input  to  the  subsequent  outcome  analysis.  More 
detailed  information  about  key  drivers  of  success  and  failure  is  pre¬ 
sented  in  Appendix  B:  Key  Drivers  of  Success  and  Failure. 


During  preparation  activities,  key  objectives  for  a  project  or  program 
are  identified.  A  key  objective  in  MAAP  is  represented  as  a  future  out¬ 
come  or  result  that  should  be  achieved  by  a  project  or  program.  The 
overall  picture  of  success  for  a  project  or  program  normally  comprises 
multiple  key  objectives.  During  outcome  analysis,  the  current  potential 
for  success  for  each  key  objective  is  assessed  in  relation  to  a  set  of  suc¬ 
cess  criteria  based  on  the  performance  characteristics  of  the  project  or 
process  (as  defined  by  the  operational  model)  and  data  about  strengths 
and  weaknesses  of  the  project  or  process. 


A  causal  analysis  identifies  the  conditions  and  circumstances  that  are 
driving  the  current  potential  for  success.  The  result  can  range  from  a 
simple  listing  of  the  key  causes  to  a  root-cause  diagram  that  depicts  the 
interrelationships  and  dependencies  among  the  conditions  and  circums¬ 
tances. 
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Uncertainty 

Analysis 


Group  Decision 
Making 


Some  degree  of  uncertainty  will  exist  with  respect  to  the  data  used  dur¬ 
ing  the  outcome  analysis.  Some  information  gaps  will  exist,  certain 
information  will  not  be  trusted  based  on  its  source,  and  other  pieces  of 
information  will  be  based  on  people’s  assumptions  or  opinions,  which 
might  prove  to  be  incorrect.  Because  of  uncertainty,  the  actual  poten¬ 
tial  for  success  for  a  key  objective  might  deviate  from  the  value  you 
determined  during  the  outcome  analysis.  An  uncertainty  analysis  is 
used  to  determine  the  uncertainty  range  for  the  current  potential  for 
success  (i.e.,  the  best-  and  worst-case  scenarios  based  on  the  degree  of 
uncertainty  inherent  in  the  distributed  project  or  process).  The  reason¬ 
ing  underlying  the  uncertainty  analysis  is  also  documented. 


When  analyzing  outcomes  in  a  group  setting,  you  can  use  techniques  to 
facilitate  decision-making  activities.  For  example,  voting  techniques, 
such  as  multi-voting,  can  help  a  group  sort  through  differences  and 
reach  consensus. 
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3.3.4  Perform  Event  Analysis  (Activity  A4) 

Introduction  This  protocol  activity  estimates  each  key  objective’s  potential  for  suc¬ 

cess  based  on  the  potential  occurrence  of  events.  These  events  are  oc¬ 
currences  that  change  the  current  state  (i.e.,  changes  the  status  quo)  for 
a  project  or  process.  The  occurrence  of  an  event  can  affect  each  key 
objective’s  potential  for  success,  either  positively  or  negatively.  In  this 
activity,  the  operational  model  provides  the  basis  for  determining  how 
events  might  affect  a  key  objective’s  potential  for  success. 


Objective  This  activity  answers  the  following  questions: 

•  Which  events  could  affect  each  key  objective’s  current  potential 
for  success? 

•  What  would  be  the  value  of  each  key  objective’s  potential  for  suc¬ 
cess  if  those  events  were  to  occur? 

•  What  conditions  and  circumstances  are  driving  the  changes  to  each 
key  objective’s  potential  for  success? 


3.3.4. 1  Dataflow 


Inputs 

01  Operational  model 
N1  Strengths  and  weaknesses 

For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 


Constraints 

01  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PR03  Assessment  plan 
PR04  Assessment  logistics 

1 

/  \ 

A4 

->■  Perform  event  t 

analysis 

V _ ^ _ 


Outputs 

For  each  key  objective: 

05  Sensitivity  to  events 

06  Causal  analysis  for  event  sensitivity 


Resources 

PROS  Trained  personnel 

PR06  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Figure  1 9:  Inputs  and  Outputs  for  Activity  A4 
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Inputs 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  the  key  objectives  of  each 
team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

N1  Strengths  and  weak¬ 
nesses 

Conditions  and  circumstances  that  are  guiding  a  project  or  process  towards 
success  (i.e.,  strengths)  and  failure  (i.e.,  weaknesses).  Examples  of  strengths 
include  a  highly  trained  workforce  and  documented  policies  and  procedures. 
Examples  of  weaknesses  include  inefficient  workflow  layout  and  inadequate 
resource  allocation. 

02  Current  potential  for 
success  (for  each  key  ob¬ 
jective) 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded.  The  current  potential  for  success  is  determined  for  each  key  ob¬ 
jective. 

03  Causal  analysis  of  cur¬ 
rent  state  (for  each  key 
objective) 

The  conditions  and  circumstances  that  are  driving  the  current  potential  for  suc¬ 
cess.  A  causal  analysis  of  the  current  state  is  developed  for  each  key  objective. 

04  Uncertainty  range  and 
rationale  (for  each  key 
objective) 

The  best-  and  worst-case  scenarios  for  the  current  potential  for  success  based 
on  the  degree  of  uncertainty  inherent  in  the  distributed  project  or  process  and 
the  justification  underlying  the  best-  and  worst-case  scenarios.  An  uncertainty 
range  is  determined  for  each  key  objective. 

Outputs 

Description 

05  Sensitivity  to  events  (for 
each  key  objective) 

The  potential  for  success  for  each  key  objective  given  the  occurrence  of  an 
event.  The  sensitivity  to  multiple  events  is  analyzed. 

06  Causal  analysis  for 
event  sensitivity  (for  each 
key  objective) 

The  conditions  and  circumstances  that  are  driving  the  potential  for  success  for 
each  key  objective  given  the  occurrence  of  an  event 

3. 3.4.2  Techniques 

T ECHNIQUES  The  techniques  employed  when  conducting  this  protocol  activity  de¬ 

pend  upon  the  knowledge,  skills,  and  abilities  of  the  people  who  are 
performing  the  assessment.  Determining  the  sensitivity  to  events  re¬ 
quires  techniques  for  projecting  what  might  happen  if  a  given  event 
were  to  occur.  In  collaborative  settings,  group  decision-making  tech¬ 
niques  can  also  be  effective. 
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Event  Analysis 


Causal  Analysis 


Group  Decision 
Making 


During  event  analysis,  the  success  potential  of  each  key  objective  is 
analyzed  for  a  range  of  events.  The  potential  for  success  is  assessed  in 
relation  to  a  set  of  success  criteria  based  on 

•  the  occurrence  of  an  event 

•  the  performance  characteristics  of  the  project  or  process  (as  de¬ 
fined  by  the  operational  model) 

•  data  about  strengths  and  weaknesses  of  the  project  or  process 

Using  the  current  potential  for  success  for  each  key  objective  as  the 
baseline  for  current  performance,  you  note  which  events  improve  the 
current  potential  for  success  and  which  diminish  it.  For  any  events  that 
have  an  impact  on  the  current  potential  for  success  for  a  key  objective, 
you  must  also  estimate  the  value  for  that  potential  for  success  if  the 
event  were  to  occur.  The  change  to  a  key  objective’s  current  potential 
for  success  due  to  the  occurrence  of  an  event  provides  a  measure  of  a 
project’s  or  process’  sensitivity  to  that  event. 


A  causal  analysis  identifies  the  conditions  and  circumstances  that  are 
driving  a  key  objective’s  potential  for  success  given  the  occurrence  of 
an  event.  When  analyzing  the  causes  of  an  event,  you  must  consider 
the  following: 

•  the  conditions  and  circumstances  that  are  exposing  the  project  or 
process  the  effects  of  each  potential  event 

•  the  conditions  and  circumstances  that  are  causing  the  potential 
event  to  improve  or  diminish  the  current  potential  for  success 

The  result  of  the  causal  analysis  can  range  from  a  simple  listing  of  the 
key  causes  to  a  root-cause  diagram  that  depicts  the  interrelationships 
and  dependencies  among  the  conditions  and  circumstances. 


When  analyzing  events  in  a  group  setting,  you  can  use  techniques  to 
facilitate  decision-making  activities.  For  example,  voting  techniques, 
such  as  multi-voting,  can  help  a  group  sort  through  differences  and 
reach  consensus. 
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3.3.5  Establish  Success  Profile  (Activity  A5) 


Introduction 


Objectives 


This  protocol  activity  generates  and  analyzes  a  success  profile  for  each 
key  objective  by 

1 .  setting  the  success  threshold 

2.  analyzing  the  current  potential  for  success,  the  uncertainty  range, 
and  the  sensitivity  to  events  in  relation  to  the  success  threshold 

3.  deciding  whether  or  not  the  success  profile  is  acceptable 

The  success  threshold  for  a  key  objective  (i.e.,  the  desired  potential  for 
success)  represents  the  goal  for  that  objective  based  on  the  input  of  key 
stakeholders. 


This  activity  answers  the  following  questions: 

•  What  is  the  desired  potential  for  success  (i.e.,  success  threshold) 
for  the  project  or  process? 

•  For  each  key  objective,  what  is  the  gap  (success  differential)  be¬ 
tween  the  success  threshold  and  the  current  potential  for  success? 

•  For  each  key  objective,  what  is  the  gap  (success  differential)  be¬ 
tween  the  success  threshold  and  the  best-  and  worst-case  scenarios 
for  the  current  potential  for  success  (i.e.,  uncertainty  range)? 

•  For  each  key  objective,  what  is  the  gap  (success  differential)  be¬ 
tween  the  success  threshold  and  the  potential  for  success  for  each 
event? 

•  What  conditions  and  potential  events  are  driving  these  gaps?  Flow? 

•  To  what  extent  are  the  current  potential  for  success,  the  uncertainty 
range,  and  the  sensitivity  to  events  for  each  key  objective  accepta¬ 
ble? 
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3.3. 5.1  Dataflow 


Inputs 

01  Operational  model 
N1  Strengths  and  weaknesses 

For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 
05  Sensitivity  to  events 
06  Causal  analysis  for  event  sensitivity 


Constraints 

Cl  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 


Outputs 

For  each  key  objective: 

07  Success  profile 


Resources 

PR05  Trained  personnel 

PR06  MAAP  assessment  procedures 

PR07  MAAP  assessment  artifacts  &  tools 


Figure  20:  Inputs  and  Outputs  for  Activity  AS 


Inputs 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  the  key  objectives  of  each 
team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

N1  Strengths  and  weak¬ 
nesses 

Conditions  and  circumstances  that  are  guiding  a  project  or  process  towards 
success  (i.e.,  strengths)  and  failure  (l.e.,  weaknesses).  Examples  of  strengths 
include  a  highly  trained  workforce  and  documented  policies  and  procedures. 
Examples  of  weaknesses  Include  inefficient  workflow  layout  and  inadequate 
resource  allocation. 

02  Current  potential  for 
success  (for  each  key  ob¬ 
jective) 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded.  The  current  potential  for  success  is  determined  for  each  key  ob¬ 
jective. 

03  Causal  analysis  of  cur¬ 
rent  state  (for  each  key 
objective) 

The  conditions  and  circumstances  that  are  driving  the  current  potential  for  suc¬ 
cess.  A  causal  analysis  of  the  current  state  is  developed  for  each  key  objective. 

04  Uncertainty  range  and 
rationale  (for  each  key 
objective) 

The  best-  and  worst-case  scenarios  for  the  current  potential  for  success  based 
on  the  degree  of  uncertainty  inherent  in  the  distributed  project  or  process  and 
the  justification  underlying  the  best-  and  worst-case  scenarios.  An  uncertainty 
range  is  determined  for  each  key  objective. 

05  Sensitivity  to  events  (for 
each  key  objective) 

The  potential  for  success  for  each  key  objective  given  the  occurrence  of  an 
event.  The  sensitivity  to  multiple  events  is  analyzed. 

06  Causal  analysis  for 
event  sensitivity  (for  each 
key  objective) 

The  conditions  and  circumstances  that  are  driving  the  potential  for  success  for 
each  key  objective  given  the  occurrence  of  an  event 
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Output 

Description 

07  Success  profile  (for 
each  key  objective) 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  for  success,  or  success  threshold 

•  the  uncertainty  range  (i.e.,  best-  and  worst-case  scenarios)  for  the  current 
potential  for  success 

•  sensitivity  to  a  range  of  events 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold,  the  uncertainty  range,  and  the  sensitivity  to  events 

A  success  profile  is  developed  for  each  key  objective. 

3. 3. 5.2  Techniques 


T ECHNIQUES  The  following  types  of  techniques  are  used  when  establishing  a  success 

profile: 

•  establishing  the  success  threshold,  using  success  criteria 

•  data  collection 

•  gap  analysis 

•  group  decision  making 

Establishing  the  The  potential  for  success  characterizes  the  likelihood,  or  probability. 
Success  T hreshold  that  the  desired  outcome  will  be  achieved  or  exceeded.  The  success 

threshold  is  the  desired,  or  target,  probability  for  the  project  or  process 
from  the  perspective  of  key  stakeholders  (e.g.,  a  15%  return  on  invest¬ 
ment,  90%  satisfied  customers,  80%  of  functional  requirements  deli¬ 
vered).  It  reflects  the  balance  between  the  stakeholders’  overall  toler¬ 
ance  for  risk  and  their  desire  for  realizing  opportunity.  Techniques  for 
establishing  the  success  threshold  enable  you  to 

•  review  the  data  that  you  collected  from  each  key  stakeholder 

•  identify  which  stakeholders  are  the  key  decision  makers  for  the 

project  or  process 

•  determine  the  decision-makers’  balance  between  the  overall  toler¬ 
ance  for  risk  and  the  desire  for  realizing  opportunity 

•  select  the  success  threshold  that  most  appropriately  reflects  the 
perspective  of  key  stakeholders 

•  confirm  the  success  threshold  with  key  stakeholders  prior  to  per¬ 
forming  the  gap  analysis,  if  needed 
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Data  Collection 


Data  Collection 

You  might  collect  all  of  the  data  you  need  to  establish  the  success  thre¬ 
shold  when  meeting  with  stakeholders  during  preparation.*"*  Alterna¬ 
tively,  you  might  get  the  information  you  need  during  Activities  A1 
and  A2.  Sometimes,  you  will  find  that  you  need  to  collect  additional 
data  when  you  are  ready  to  set  the  success  threshold  during  this  proto¬ 
col  activity. 

Gap  Analysis 

Gap-analysis  techniques  are  used  when  analyzing  the  success  profile. 
Simple  gap  analysis  techniques  are  used  to  determine  whether  a  key 
objective’s  current  potential  for  success  is  acceptable.  Other  gap- 
analysis  techniques  also  determine  which  conditions  are  contributing  to 
the  gap  and  how. 

Group  Decision 

Making 

When  analyzing  the  success  profile  in  a  group  setting,  you  can  use 
techniques  to  facilitate  decision-making  activities.  For  example,  voting 
techniques,  such  as  multi-voting,  can  help  a  group  sort  through  differ¬ 
ences  and  reach  consensus. 

”  The  success  threshold  must  be  set  by  the  time  this  protocol  activity  is  performed.  However,  you  can  set  the 
success  threshold  earlier  in  the  assessment,  for  example  during  the  Phase  1  preparation  activities,  based  on  in¬ 
formation  gathered  from  senior  managers  and  others. 
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3.3.6  Determine  Next  Steps  (Activity  A6) 


iNTRODUCTION  This  protocol  activity  identifies  the  steps  or  actions  that  will  be  imple¬ 

mented  after  the  assessment  is  complete.  The  results  of  this  activity 
serve  as  a  bridge  between  the  MAAP  assessment  and  any  follow-on, 
detailed  strategy  development  and  planning  activities.  All  actions,  or 
next  steps,  identified  during  this  protocol  activity  should  be  at  an  ap¬ 
propriate  level  of  detail  based  on  the  goals  of  the  assessment,  depth 
and  breadth  of  the  data  collected,  analysis  algorithm  used,  knowledge, 
skills,  and  abilities  of  the  people  conducting  the  assessment,  and  ex¬ 
pectations  of  stakeholders.*^ 


Objective  This  activity  answers  the  following  questions: 


•  What  steps  or  actions  need  to  be  taken  after  the  assessment  is 
complete? 

•  Who  is  responsible  for  each  action? 

•  By  when  must  each  action  be  completed? 

3. 3. 6.1  Dataflow 

Constraints 

C1  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PROS  Assessment  plan 
PR04  Assessment  logistics 

Inputs 

01  Operational  model 
N1  Strengths  and  weaknesses 

For  each  key  objective: 

02  Current  potential  for  success 
03  Causal  analysis  of  current  state 
04  Uncertainty  range  &  rationale 
05  Sensitivity  to  events 
06  Causal  analysis  for  event  sensitivity 
07  Success  profile 

PROS  Trained  personnel 
PR06  MAAP  assessment  procedures 
PR07  MAAP  assessment  artifacts  &  tools 


A6 

Determine 
next  steps 


Output 

08  Next  steps 


Resources 


Figure  21:  Inputs  and  Outputs  for  Activity  A6 


The  results  of  this  protocol  activity  can  range  from  a  simple  set  of  recommendations  or  list  of  action  items  to  a 
detailed  plan  that  includes  resource  estimates,  budget,  and  schedule. 
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Inputs 

Description 

01  Operational  model 

A  detailed,  descriptive  model  of  the  distributed  project  or  process  being  as¬ 
sessed.  At  a  minimum,  an  operational  model  must  document  the  following: 

•  the  key  objectives  of  the  project  or  process  and  the  key  objectives  of  each 
team  or  group 

•  all  key  activities  performed  by  each  team  or  group 

•  the  sequence  in  which  activities  are  performed 

•  the  work  products  produced  by  each  key  activity 

N1  Strengths  and  weak¬ 
nesses 

Conditions  and  circumstances  that  are  guiding  a  project  or  process  toward 
success  (i.e.,  strengths)  or  failure  (i.e.,  weaknesses).  Examples  of  strengths 
include  a  highly  trained  workforce  and  documented  policies  and  procedures. 
Examples  of  weaknesses  include  inefficient  workflow  layout  and  inadequate 
resource  allocation. 

02  Current  potential  for 
success  (for  each  key  ob¬ 
jective) 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded.  The  current  potential  for  success  is  determined  for  each  key  ob¬ 
jective. 

03  Causal  analysis  of  cur¬ 
rent  state  (for  each  key 
objective) 

The  conditions  and  circumstances  that  are  driving  the  current  potential  for  suc¬ 
cess.  A  causal  analysis  of  the  current  state  is  developed  for  each  key  objective. 

04  Uncertainty  range  and 
rationale  (for  each  key 
objective) 

The  best-  and  worst-case  scenarios  for  the  current  potential  for  success  based 
on  the  degree  of  uncertainty  inherent  in  the  distributed  project  or  process  and 
the  justification  underlying  the  best-  and  worst-case  scenarios.  An  uncertainty 
range  is  determined  for  each  key  objective. 

05  Sensitivity  to  events  (for 
each  key  objective) 

The  potential  for  success  for  each  key  objective  given  the  occurrence  of  an 
event.  The  sensitivity  to  multiple  events  is  analyzed. 

06  Causal  analysis  for 
event  sensitivity  (for  each 
key  objective) 

The  conditions  and  circumstances  that  are  driving  the  potential  for  success  for 
each  key  objective,  given  the  occurrence  of  an  event 

07  Success  profile  (for 
each  key  objective) 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  for  success,  or  success  threshold 

•  the  uncertainty  range  (i.e.,  best-  and  worst-case  scenarios)  for  the  current 
potential  for  success 

•  sensitivity  to  a  range  of  events 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold,  the  uncertainty  range,  and  the  sensitivity  to  events 

A  success  profile  is  developed  for  each  key  objective. 

Output 

Description 

08  Next  steps 

Actions  to  be  taken  after  the  assessment  is  complete 
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3. 3. 6.2  Techniques 


Techniques 

Several  types  of  techniques  can  be  used  when  you  are  determining 
what  approach  to  take  after  the  assessment,  including 

•  action  planning 

•  brainstorming 

•  group  decision  making 

Action  Planning 

Action  planning  is  a  basic  technique  for  determining  how  to  proceed 
after  a  MAAP  assessment  is  complete.  When  performing  this  tech¬ 
nique,  you 

1 .  identify  a  candidate  list  of  actions,  or  next  steps  (often  using 
brainstorming  techniques) 

2.  determine  which  of  the  candidate  actions  will  be  implemented 
after  the  assessment  is  complete 

The  results  of  action  planning  lay  the  groundwork  for  subsequent  im¬ 
provement  activities. 

Brainstorming 

Brainstorming  is  a  basic  technique  for  generating  ideas.  It  can  be  used 
to  identify  a  candidate  list  of  actions  for  maintaining  or  improving  the 
current  potential  for  success.  Many  variants  of  brainstorming  exist  and 
can  be  used  when  performing  this  protocol  activity. 

Group  Decision 

Making 

When  selecting  an  appropriate  set  of  next  steps,  you  can  use  techniques 
to  facilitate  decision-making  activities.  For  example,  voting  tech¬ 
niques,  such  as  multi-voting,  can  help  a  group  sort  through  differences 
and  reach  consensus. 
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3.4  COMPLETE  POST-ASSESSMENT  ACTIVITIES  (PHASE  3) 


Introduction  Phase  3  conveys  the  results  of  the  MAAP  assessment  to  key  stakehold¬ 

ers  and  identifies  actions  that  can  improve  the  efficiency  and  effective¬ 
ness  of  the  MAAP  assessment.  The  objective  when  communicating 
assessment  results  to  stakeholders  is  to  present  findings  in  a  format  that 
meets  their  needs  and  requirements.  Different  formats  might  be  needed 
to  communicate  results  to  different  types  of  stakeholders. 

A  postmortem  is  used  to  identify  and  document  ways  in  which  the 
MAAP  assessment  can  be  improved.*^  Updates  and  improvements  to 
MAAP  assessment  procedures,  artifacts,  tools,  and  training  are  made 
as  appropriate. 


Objectives  This  protocol  phase  answers  the  following  questions: 

•  Who  else  needs  to  know  the  results  of  the  assessment?*’ 

•  What  information  does  each  stakeholder  need? 

•  How  should  information  be  communicated  to  each  stakeholder? 

•  What  lessons  were  learned  when  preparing  for  the  assessment? 

•  What  lessons  were  learned  when  conducting  the  assessment? 

•  How  do  the  assessment  procedures,  artifacts,  tools,  and  training 
need  to  be  updated  or  improved? 


Dataflow 


The  following  diagram  highlights  the  dataflow  for  this  protocol  phase. 


Constraint 

C1  Assessment  constraints 


Inputs 

PAI1  MAAP  assessment  results  &  plans 
PAI2  MAAP  assessment  procedures, 
artifacts,  tools,  &  training 


Phase  3 
Complete  Post- 
Assessment 
Activities 


Outputs 

PA01  Communicated  assessment  results 
PA02  Lessons  learned 
PA03  Updates  to  MAAP  assessment 
procedures,  artifacts,  tools,  &  training 


Resources 

R5  Experienced  personnel 
R6  Post-assessment  procedures 
R7  Post-assessment  artifacts  &  tools 


Figure  22:  Dataflow  for  MAAP  Phase  3 


Postmortems  are  usually  conducted  after  a  given  assessment.  However,  they  can  also  be  held  on  a  periodic 
basis  if  multiple  assessments  are  planned. 


17 


Requirements  for  communicating  assessment  results  are  part  of  the  assessment  plan  that  is  developed  in 
Phase  1.  These  requirements  are  revisited  in  Phase  3  and  are  revised  when  appropriate  (e.g.,  if  new  stake¬ 
holders  are  identified  during  the  assessment). 
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Inputs 


The  following  inputs  are  required  by  the  activities  performed  during 
this  protocol  phase. 


Type 

Description 

PAM  MAAP  assessment 
results  and  plans 

All  outputs  produced  by  the  MAAP  assessment,  including  findings  and  assess¬ 
ment  data,  as  well  as  plans,  budget,  and  schedule  for  conducting  the  assess¬ 
ment 

PAI2  MAAP  assessment 
procedures,  artifacts,  tools, 
and  training 

Supporting  materials  used  to  conduct  a  MAAP  assessment,  including  proce¬ 
dures,  worksheets,  databases,  and  training  artifacts 

Constraint  The  following  constraint  affects  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

C1  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

Resources  The  following  resources  support  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

R5  Experienced  personnel 

People  who  are  experienced  in  all  phases  of  a  MAAP  assessment 

R6  Post-assessment  pro¬ 
cedures 

Documentation  that  describes  how  to  conduct  post-assessment  activities 

R7  Post-assessment  arti¬ 
facts  and  tools 

Templates,  worksheets,  standard  presentations,  automated  tools,  and  data¬ 
bases  needed  to  conduct  post-assessment  activities 

Outputs  The  following  outputs  are  produced  by  the  activities  performed  during 

this  protocol  phase. 


Type 

Description 

PA01  Communicated  as¬ 
sessment  results 

Assessment  results  that  have  been  conveyed  to  key  stakeholders.  Results 
include 

•  operational  model 

•  success  profile  for  each  key  objective  of  the  project  or  process 

•  actions  that  need  to  be  implemented  after  the  assessment  is  complete 

•  supporting  data  as  appropriate 

56  I  CMU/SEI-2008-TN-011 


Type 

Description 

PA02  Lessons  learned 

Knowledge  gained  by  conducting  a  MAAP  assessment  that  can  be  used  to 
modify  and  improve  future  MAAP  assessments 

PA03  Updates  to  MAAP 
assessment  procedures, 
artifacts,  tools,  and  training 

Any  changes,  based  on  lessons  learned,  to  MAAP  assessment  procedures, 
artifacts,  tools,  and  training  intended  to  improve  the  efficiency  and  effectiveness 
of  future  MAAP  assessments 

Key  Activities  The  following  table  highlights  the  activities  performed  during  this  pro¬ 

tocol  phase/^ 


Activity 

Description 

Communicate  results 

Convey  the  results  of  the  MAAP  assessment  to  key  stakeholders 

Conduct  postmortem  of  the 
MAAP  assessment 

Conduct  one  or  more  meetings  to  identify  the  strengths  and  weaknesses  of  the 
MAAP  assessment  and  document  modifications  and  improvement  to  the  MAAP 
assessment  process 

Implement  improvements  to 
the  MAAP  assessment 

process 

Make  changes,  based  on  lessons  learned,  to  the  MAAP  assessment  process, 
including  updating  procedures,  artifacts,  tools,  and  training  as  appropriate 

Detailed  descriptions  of  Phase  3  activities  are  not  provided  in  this  document. 
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4  Summary  and  Further  Work 


Mission  Success 

Research  and 

Development 

In  2006,  the  Carnegie  Mellon®  Software  Engineering  Institute  char¬ 
tered  the  Mission  Success  in  Complex  Environments  (MSCE)  project 
to  advance  the  risk-management  state-of-the-practice.  A  key  part  of 
this  project  is  the  development  of  MOSAIC —  a  suite  of  risk-based 
methods  for  assessing  and  managing  complex  projects  and  processes. 
MAAP  is  the  second  MOSAIC  assessment  protocol  to  be  published; 
the  Mission  Diagnostic  Protocol  (MDP)  was  the  first.  Please  refer  to 
the  MSCE  web  site  for  current  information: 

http://www.sei.cmu.edu/msce/ 

MAAP  Assessments 

MAAP  is  a  risk-based,  systematic  approach  for  assessing  the  potential 
for  success  in  distributed  environments  and  provides  a  foundation  for 
collaboratively  managing  the  potential  for  success  in  distributed  envi¬ 
ronments.  It  is  designed  to  help  people  analyze  tradeoffs  and  make 
better  decisions  in  situations  that  have  a  high  degree  of  uncertainty.  An 
operational  model  of  the  project  or  process  is  used  as  the  foundation 
for  complex  analyses  of  current  conditions  and  relevant  events  to  de¬ 
termine  the  degree  of  success  possible  under  a  variety  of  circums¬ 
tances.  It  is  a  time-  and  resource-intensive  approach  that  provides  a 
rich,  in-depth  set  of  information  about  the  project  or  process.  As  such, 
it  should  only  be  undertaken  with  careful  consideration  of  the  costs 
and  requirements. 

MAAP  Pilots 

MAAP  was  designed  for  use  in  many  different  domains  and  types  of 
problems.  To  date,  MAAP  has  been  piloted  in  the  domain  of  cyber¬ 
security  incident  response,  although  some  aspects  have  been  piloted  in 
software  development  and  deployment. 

Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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MOSAIC 

Management 

Approach 


Figure  23: 


MOSAIC  requires  establishing  and  maintaining  a  reasonable  degree  of 
confidence  that  objectives  will  be  achieved  successfully.  Figure  23 
depicts  the  key  activities  performed  when  using  MOSAIC  to  manage  a 
project  or  process.  Notice  that  an  assessment  is  a  key  activity  of  this 
approach.  However,  assessing  the  current  potential  for  success  (e.g., 
by  performing  a  MAAP  assessment)  is  just  one  part  of  the  broader 
management  approach.  Additional  follow-on  activities  are  needed  to 
help  ensure  that  the  desired  outcome  will  be  achieved. 
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MOSAIC 

Management 

Activities 


Future  MAAP 
Development 


Future  MSCE 
Research  Directions 


Research  Goal 


As  illustrated  in  Figure  23,  MOSAIC  requires  completing  the  follow¬ 
ing  key  activities: 

•  Assess — determine  the  current  potential  for  success  in  relation  to 
its  success  threshold 

•  Plan — develop  a  detailed  action  plan  for  maintaining  or  improving 
the  potential  for  success  over  time 

•  Implement — execute  plans  as  defined 

•  Track — ^monitor  the  status  of  plan  milestones  and  measures  of  plan 
effectiveness 

•  Control — ^make  adjustments  to  plans  when  appropriate 

MAAP  enables  you  to  assess  the  current  potential  for  success  as  well 
as  the  potential  for  success  for  different  outcomes  and  during  different 
events.  In  addition,  it  provides  a  solid  foundation  for  planning  and  im¬ 
provement  activities. 


MAAP  is  an  important  piece  of  research  because  it  provides  a  founda¬ 
tion  for  future  research  and  development  activities  related  to 
MOSAIC.  We  intend  to  continue  piloting  MAAP  in  different  venues. 
We  also  intend  to  provide  additional  information,  such  as  training,  for 
MAAP. 


We  intend  to  continue  developing  the  MOSAIC  suite  of  assessment 
and  management  methods.  The  early  focus  of  MOSAIC  research  has 
been  on  assessing  the  potential  for  success.  Future  research  will  focus 
on  developing  approaches  for  managing  the  potential  for  success  over 
time. 


Overall,  the  main  goal  of  our  research  is  to  transform  risk  management 
from  a  hazard-driven  discipline  to  a  success-  and  opportunity-driven 
discipline.  Our  work  with  MAAP  is  a  step  toward  achieving  that  goal. 


60  I  CMU/SEI-2008-TN-011 


Appendix  A:  Risk  Management  Concepts 


Introduction 

This  appendix  presents  a  brief  overview  of  traditional  risk  manage¬ 
ment  concepts.  The  ideas  presented  here  describe  the  current  state  of 
the  practice  for  risk  management  and  provide  the  foundational  basis 
for  the  research  featured  in  this  document. 

Multiple  Contexts 

OF  Risk  Management 

The  term  risk  is  used  universally,  but  different  audiences  often  attach 
different  meanings  to  it  [Kloman  90] .  In  fact,  the  details  about  risk  and 
how  it  supports  decision  making  depend  upon  the  context  in  which  it 
is  applied  [Charette  90].  For  example,  safety  professionals  view  risk 
management  in  terms  of  reducing  the  number  of  accidents  and  injuries. 
A  hospital  administrator  views  risk  as  part  of  the  organization’s  quali¬ 
ty  assurance  program,  while  the  insurance  industry  relies  on  risk  man¬ 
agement  techniques  when  setting  insurance  rates.  Each  industry  thus 
uses  a  definition  that  is  uniquely  tailored  to  its  context.  No  universally 
accepted  definition  of  risk  exists. 

Three 

Characteristics  of 

Risk 

Whereas  specific  definitions  of  risk  might  vary,  a  few  characteristics 
are  common  to  all  definitions.  For  risk  to  exist  in  any  circumstance, 
the  following  three  conditions  must  be  satisfied  [Charette  1990]: 

1 .  The  potential  for  loss  must  exist. 

2.  Uncertainty  with  respect  to  the  eventual  outcome  must  be 

*  19 

present. 

3.  Some  choice  or  decision  is  required  to  deal  with  the  uncertainty 
and  potential  for  loss. 

Three  Conditions  of 

Risk 

The  three  characteristics  can  be  used  to  forge  a  very  basic  definition  of 
the  word  risk.  Most  definitions  focus  on  the  first  two  conditions — loss 
and  uncertainty — because  they  are  the  two  measurable  aspects  of  risk. 
Thus,  the  essence  of  risk,  no  matter  what  the  domain,  can  be  succinctly 
captured  by  the  following  definition:  Risk  is  the  possibility  of  suffering 
loss  [Dorofee  1996]. 

Some  researchers  separate  the  concepts  of  certainty  (the  absence  of  doubt),  risk  (where  the  probabilities  of 
alternative  outcomes  are  known),  and  uncertainty  (where  the  probabilities  of  possible  outcomes  are  unknown). 
However,  because  uncertainty  is  a  fundamental  attribute  of  risk,  we  do  not  differentiate  between  decision  mak¬ 
ing  under  risk  and  decision  making  under  uncertainty  in  this  technical  note. 
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Speculative  and 
Hazard  Risk 


Figure  24: 

Speculative  Risk 
Example:  Gambling 


Sometimes  a  situation  presents  an  opportunity  for  gain  as  well  as  the 
potential  for  loss.  In  other  instances,  only  the  potential  for  loss  exists. 
Because  of  this  difference,  risk  can  thus  be  further  subdivided  into  two 
types:  speculative  risk  and  hazard  risk  [Young  2001].  Figure  24  graph¬ 
ically  illustrates  the  difference  between  speculative  and  hazard  risk. 

With  speculative  risk  you  might  realize  a  gain,  which  can  improve 
your  current  situation  relative  to  the  status  quo.  At  the  same  time,  you 
might  experience  a  loss,  which  can  make  your  situation  worse  than  it 
is  at  present.  In  contrast,  hazard  risk  provides  no  opportunity  to  im¬ 
prove  upon  the  current  situation;  it  only  brings  the  potential  for  loss. 


Speculative  and  Hazard  Risks 


Gambling  is  an  example  of  taking  a  speculative  risk.  When  you  place  a 
bet,  you  must  balance  the  potential  for  gain  against  the  potential  for 
loss.  You  weigh  the  possibility  of  gaining  additional  money  against  the 
prospect  of  losing  the  funds  you  wagered.  When  you  gamble,  your 
objective  is  to  increase  your  wealth  in  relation  to  the  status  quo,  and 
you  are  willing  to  put  your  finances  at  risk  for  the  opportunity  to  make 
money. 
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Speculative  Risk 
Example:  Business 
Risk 


Hazard  Risk 
Example:  Security 


Speculative  Risk 
Example:  Security 


Business  risk  is  another  example  of  speculative  risk.  When  managers 
invest  organizational  assets,  they  must  balance  the  risk  of  investing 
organizational  capital  against  the  potential  return  on  that  investment. 
From  an  economic  perspective,  as  an  organization’s  risk  increases,  its 
potential  return  on  investment  had  better  increase  correspondingly. 
Management  should  never  take  on  additional  risk  unless  the  potential 
for  increased  profits  also  exists.  The  balance  of  risk  and  opportunity 
drives  all  business  decisions,  which  makes  business  risk  speculative. 


Consider  how  security  can  be  viewed  as  a  hazard  risk.  Imagine  that 
you  are  concerned  about  protecting  valuables  that  are  stored  in  your 
home.  Your  main  objective  in  this  example  is  to  ensure  that  none  of 
the  valuables  in  your  residence  is  removed  without  your  knowledge 
and  permission.  After  evaluating  how  well  your  valuables  are  pro¬ 
tected,  you  might  decide  to  install  a  security  system  in  your  residence 
to  make  it  more  difficult  for  a  thief  to  break  in  and  steal  your  valua¬ 
bles.  Notice  that  the  objective  in  this  example,  by  definition,  restricts 
the  focus  of  risk  on  the  potential  for  loss.  In  the  most  favorable  of  cir¬ 
cumstances,  you  only  keep  what  you  already  possess.  There  is  no  po¬ 
tential  for  gain. 


Now  consider  the  same  example  when  viewed  from  another  perspec¬ 
tive.  In  this  instance,  you  would  like  to  gain  peace  of  mind  by  prevent¬ 
ing  unsavory  characters  from  gaining  entrance  to  your  house.  Your 
objective  to  feel  more  secure  defines  the  context  in  which  you  view 
risk.  After  analyzing  the  situation,  you  might  decide  to  install  a  securi¬ 
ty  system  in  your  residence  to  make  it  more  difficult  for  someone  to 
break  in.  You  might  reason  that  the  added  protection  will  make  you 
feel  more  secure  and  help  you  gain  the  peace  of  mind  you  seek. 

In  this  example,  you  are  willing  to  invest  money  in  a  security  system 
to  provide  yourself  an  opportunity  feel  more  secure.  The  security  risk 
in  this  example  is  speculative  because  it  balances  your  tolerance  for 
risk  (i.e.,  the  amount  of  money  you  are  willing  to  invest  in  a  security 
system)  with  your  desire  to  realize  an  opportunity  (i.e.,  gaining  peace 
of  mind). 
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Importance  of  The  two  security  examples  illustrate  how  the  same  situation  can  be 

Context  viewed  as  a  hazard  risk  in  one  context  and  a  speculative  risk  in  anoth¬ 

er.  A  risk  therefore  is  classified  as  speculative  or  hazard  based  on  the 
context  in  which  it  is  viewed.  The  notion  of  explicitly  establishing  the 
context  in  which  you  analyze  and  manage  risk  is  vitally  important  to 
ensure  that  you  make  appropriate  choices  about  how  you  manage  your 
risk. 


Five  Common  All  forms  of  risk,  whether  they  are  classified  as  speculative  or  hazard 

Elements  risk,  comprise  common  elements.  This  notion  is  illustrated  in  Figure 

25,  which  highlights  the  following  five  common  elements  of  risk:  (1) 
context,  (2)  execution,  (3)  conditions,  (4)  potential  events,  and  (5) 
range  of  potential  outcomes. 


Context 


Conditions 


Potential 

events 


Execution 


\ 

Range  of  I 
potential 
outcomes  | 

/ 


Figure  25:  Common  Elements  of  Risk 


Context  Context  provides  the  background,  situation,  or  environment  in  which  a 

project  or  process  is  executed.  It  generally  includes  the  key  objectives 
being  pursued  as  well  as  stakeholders’  expectations  for  those  objec¬ 
tives.^”  It  defines  the  picture  of  success  for  a  given  set  of  objectives 
and  provides  the  lens  through  which  all  potential  outcomes  are  viewed 
and  interpreted.  Defining  the  context  is  thus  an  essential  first  step 
when  managing  any  type  of  risk. 


Stakeholders  include  all  interested  parties,  customers,  and  suppliers,  both  internal  and  external  to  an  organiza¬ 
tion. 
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Example:  Project 

Management 

Context 

Assume  that  you  are  a  project  manager  who  is  overseeing  the  devel¬ 
opment  of  a  software-intensive  system.  Suppose  that  these  are  the 
most  important  objectives  to  you:  product,  cost,  and  schedule.  These 
objectives  indicate  that  you  are  focused  on  developing  a  fully  func¬ 
tional  system  on  time  and  within  budget.  Now,  suppose  that  stake¬ 
holders  (such  as  senior  managers  in  your  organization)  are  very  con¬ 
cerned  about  cost  overrans  and  have  made  it  clear  that  the  project 
cannot  exceed  its  budget.  As  a  result,  the  cost  objective  becomes  your 
primary  objective  among  the  three,  and  your  tolerance  for  cost  risk  is 
low. 

Your  decisions  will  be  driven  by  your  low  tolerance  for  cost  overruns. 
When  you  are  forced  to  make  tradeoffs,  unacceptable  outcomes  related 
to  cost  will  have  a  greater  influence  than  those  related  to  product  and 
schedule  objectives. 

Foundation  of  Risk 

Management 

The  context  in  the  above  example  has  been  defined  by  three  project 
objectives  and  the  expectations  related  to  those  objectives.  Without 
setting  an  appropriate  context,  you  cannot  definitively  determine  how 
to  gauge  the  potential  for  success  or  how  to  assess  any  given  outcome. 
Context  thus  forms  the  underlying  foundation  when  managing  risk. 

Execution 

Execution  describes  what  must  be  done  to  achieve  a  set  of  objectives. 
With  respect  to  a  project  or  process,  execution  refers  to  the  activities 
that  are  performed  when  working  toward  the  objectives. 

Conditions 

Conditions  define  the  circumstances  that  directly  or  indirectly  influ¬ 
ence  execution  and  drive  an  outcome  toward  success  or  failure.  As  a 
project  or  process  is  executed,  these  conditions  affect  the  eventual  out¬ 
come.  In  some  instances,  conditions  directly  influence  the  outcome; 
while  in  others,  they  indirectly  affect  the  outcome  by  creating  expo¬ 
sure  to  negative  or  positive  events. 

Example:  Conditions 

THAT  Directly 

Influence  an 

Outcome 

Consider  an  example  where  a  team  is  developing  a  software-intensive 
system.  Suppose  that  the  following  condition  is  present:  team  mem¬ 
bers  have  not  previously  worked  with  the  design  language  being  used 
on  the  project.  This  could  cause  them  to  make  mistakes  or  take  more 
time  when  working  on  tasks,  driving  product,  cost,  and  schedule  ob¬ 
jectives  toward  one  or  more  undesired  outcomes.  Here,  the  condition 
has  a  direct  influence  on  the  eventual  outcome. 
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Exposing  Conditions 

Conditions  that  expose  a  project  or  process  to  the  effects  of  events  that 
might  (or  might  not)  occur  are  called  exposing  conditions.  During 
normal  day-to-day  operations,  these  conditions  lie  dormant  and  do  not 
produce  any  visible  effect  on  results.  However,  certain  events  in  com¬ 
bination  with  exposing  conditions  can  influence  the  expected  out- 
21 

come. 

Potential  Events 

A  potential  event  is  an  unpredictable  occurrence  that  combines  with 
one  or  more  exposing  conditions  to  affect  performance  and  thus  drive 
the  outcome  toward  success  or  failure. 

Example:  Potential 

Events  and  Exposing 

Conditions 

A  computer  vims  is  a  program  that  is  designed  to  exploit  certain  ex¬ 
posing  conditions  (called  vulnerabilities)  and  infect  computers  causing 
them  to  act  erratically.  People  with  malicious  intent  design  these  pro¬ 
grams  with  the  ultimate  goal  of  wreaking  havoc  throughout  the  busi¬ 
ness  community,  such  as  degrading  the  performance  of  computers  and 
networks  or  rendering  them  unavailable  for  use.  If  a  work  process  is 
highly  dependent  on  the  availability  of  computers  and  networks  that 
become  infected,  production  can  be  temporarily  halted,  which  can  lead 
to  an  undesirable  outcome. 

Notice  that  the  condition,  the  system’s  vulnerability,  poses  no  threat  to 
production  during  normal  operations.  It  takes  an  unpredictable  event, 
the  proliferation  of  a  computer  vims,  for  damage  to  occur.  This  partic¬ 
ular  condition  only  affects  the  process’  outcome  when  a  relevant  event 

occurs. 

Range  of  Potential 

Outcomes 

The  range  of  potential  outcomes  defines  the  set  of  possible  results  that 
can  be  achieved  when  executing  a  project  or  process.  Some  outcomes 
will  be  considered  to  be  acceptable,  while  others  will  be  viewed  as 
unacceptable. 

Events  can  have  a  positive  or  negative  effect  on  the  outcome  depending  on  the  specific  nature  of  the  event.  For 
example,  an  increase  in  funding  would  likely  be  perceived  as  a  positive  event  that  might  put  a  project  in  better 
position  for  success.  On  the  other  hand,  a  decrease  in  funding  would  likely  be  perceived  as  a  negative  event 
that  might  adversely  affect  a  project’s  outcome. 

Undesirable  from  the  business’  perspective,  that  is.  From  the  virus  developer’s  perspective,  this  would  be  con¬ 
sidered  a  successful  outcome. 
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Traditional  Risk 

Management 

Approaches 


Most  risk-management  approaches,  when  applied  to  projects  and 
processes,  have  traditionally  assumed  a  hazard  view  of  risk.  From  the 
hazard  perspective,  a  risk  is  viewed  as  a  potential  obstacle  that  can 
interfere  with  positive  momentum  or  progress,  and  a  threat  is  defined 
as  a  condition  or  event  that  could  lead  to  a  risk  [Alberts  2005].  When 
viewed  from  this  perspective,  traditional  risk  management  focuses  on 
reducing  or  eliminating  obstacles  that  might  interfere  with  momentum 
or  progress.  In  addition,  traditional  risk  management  approaches  have 
not  considered  multiple  organizations;  they  focus  within  an  organiza¬ 
tion  and  locally  optimize  risk  for  that  organization. 


Focus  ON  Single  Traditional  risk-management  approaches,  when  applied  to  projects  and 

Conditions  or  processes,  focus  on  individual  conditions  or  potential  events.  A  risk 

Events  analysis  is  then  used  to  estimate  the  potential  consequence  triggered 

by  each  condition  or  event. 


Risk  Statement  A  risk  is  normally  represented  using  a  linear  cause-and-effect  pair  that 

conveys  two  key  pieces  of  information:  (1)  the  threat  (i.e.,  condition  or 
potential  event)  that  is  causing  concern  and  (2)  the  potential  conse¬ 
quences  of  that  threat  [Gluch  1994].  Each  cause-and-effect  pair,  or  risk 
statement,  can  be  viewed  as  a  scenario  that  documents  the  potential 
loss  triggered  by  a  given  condition  or  event.  Figure  26  illustrates  the 
notion  of  multiple  risks  that  can  affect  a  project  or  process.  The  list  of 
risks  becomes  the  focal  point  of  risk  management  activities  in  tradi¬ 
tional  approaches. 


Figure  26:  Cause  and  Effect  Risk  Statement 


SOFTWARE  ENGINEERING  INSTITUTE  |  67 


Basic  Risk 

Management 

Activities 


Traditional  risk-management  approaches  generally  require  people  to 
conduct  the  following  types  of  activities: 

•  identify  risks  that  can  lead  to  loss 

•  prioritize  the  list  of  risk  statements  based  on  objectives,  require¬ 
ments,  and  constraints 

•  develop  mitigation  plans  for  the  highest  priority  risks 

•  implement  the  mitigation  plans  as  defined 

•  track  the  status  of  mitigation  plan  milestones  and  measures  of  ef¬ 
fectiveness 

•  make  adjustments  to  mitigation  plans  when  appropriate 
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Appendix  B:  Key  Drivers  of  Success  and  Faiiure^^ 


What  is  a  Driver? 

Drivers  are  characteristics  of  a  project  or  process  that  are  essential  for 
achieving  its  objectives.  Each  individual  driver  has  a  strong  influence 
on  the  ultimate  outcome,  or  result.  The  cumulative  effects  of  all  drivers 
can  be  analyzed  to  determine  whether  a  project  or  process  has  suffi¬ 
cient  momentum  toward  its  objectives.  Establishing  the  effects  of  driv¬ 
ers  is  crucial  when  analyzing  the  potential  for  success. 

Success  and  Failure 

Drivers 

In  MOSAIC,  each  driver  can  be  worded  as  a  success  or  failure  driver. 
Consider  the  following  example:  Task  execution  is  effective  and  effi¬ 
cient.  Here,  the  implication  is  that  people  have  sufficient  capability  to 
complete  their  assigned  tasks.  This  is  a  positive  characteristic  of  a 
project  or  process  that  helps  enhance  its  potential  for  success,  which 
makes  it  a  success  driver. 

Further,  MOSAIC  assessments  determine  which  drivers  from  a  set  are 
guiding  a  project  toward  a  successful  outcome  and  which  are  not. 

When  a  given  driver  does  not  have  a  positive  influence  on  a  project  or 
process,  it  is  acting  as  a  failure  driver.  Here,  the  driver  is  reducing  the 
momentum  toward  achieving  objectives  and  making  an  unsuccessful, 
or  failed,  outcome  more  likely.  For  example,  if  people  do  not  have  suf¬ 
ficient  capability  to  complete  their  assigned  tasks,  the  success  potential 
of  the  project  or  process  is  adversely  affected. 

For  a  more  detailed  explanation  of  drivers  and  how  to  evaluate  them,  see  Alberts,  Christopher,  Dorofee,  Aud¬ 
rey,  &  Marino,  Lisa.  Mission  Diagnostic  Protocol:  a  Risk-Based  Approach  to  Assessing  the  Potential  for  Suc¬ 
cess  (CMU/SEI-2007-TR-023).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University, 
2007.  http://www.sei.cmu.edu/publications/documents/07.reports/07tr023.html 
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Driver  Questions  Each  driver  is  characterized  as  a  yes/no  question,  where  an  answer  of 

yes  denotes  a  success  driver  and  an  answer  of  no  denotes  a  failure  driv¬ 
er.  Wording  drivers  as  either  success  or  failure  driver  questions  is  es¬ 
sentially  a  preference  of  the  analysts.  Examples  of  driver  questions 
used  in  MOSAIC  assessments  include 

•  Are  project  goals  realistic  and  well  articulated? 

•  Are  customer  requirements  and  needs  well  understood? 

•  Are  organizational  and  political  conditions  impeding  completion 
of  project  activities? 

Note  the  last  example  is  worded  as  a  failure  driver. 

Example  Driver  Set  Figure  27  below  illustrates  a  set  of  driver  questions  worded  as  success 

drivers.  The  driver  questions  can  be  embodied  in  surveys  or  used  as 
interview  questions  to  support  data  gathering  efforts. 


Driver  Questions 

1 .  Are  project  goals  realistic  and  well-articulated? 

2.  Are  communication  and  information  sharing  about  project  activities  effective? 

3.  Are  customer  requirements  and  needs  well  understood? 

4.  Are  organizational  and  political  conditions  facilitating  completion  of  project  activities? 

5.  Is  the  project  plan  sufficient? 

6.  Does  project  management  facilitate  execution  of  tasks  and  activities? 

7.  Is  task  execution  efficient  and  effective? 

8.  Is  staffing  sufficient  to  execute  all  project  activities? 

9.  Are  the  technological  and  physical  Infrastructures  adequate  to  support  all  project  activities? 

10.  Are  changing  circumstances  and  unpredictable  events  effectively  managed? 

Figure  27:  Example  Set  of  Driver  Questions 
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Using  Drivers  in 
MAAP 


Tailoring  Drivers  to 
Reflect  Key 
Characteristics  of 
Success 


In  MAAP,  a  considerable  amount  of  data  is  collected  when  developing 
the  operational  model  during  Activity  A1  and  when  identifying 
strengths  and  weaknesses  in  Activity  A2.  A  preliminary  data  analysis 
using  an  appropriate  driver  set  can  help  focus  subsequent  data  analysis 
activities. 


The  driver  set  should  be  tailored  for  each  specific  context  because  it  is 
essential  that  drivers  provide  meaningful  information  about  a  project  or 
process.  A  generic  set  of  drivers,  such  as  those  featured  in  Figure  27, 
can  be  used  as  a  starting  point  for  preliminary  data  analysis  in  MAAP. 
However,  you  need  to  ensure  that  the  driver  set  used  reflects  the  key 
characteristics  that  define  success  for  that  project  or  process. 

When  you  tailor  drivers  for  a  MAAP  assessment,  you  need  to  make 
sure  that  the  driver  set  minimally  addresses  the  following  aspects  of  a 
project  or  process: 

•  project  or  process  objectives,  including  technical,  funding  and 
schedule  objectives 

•  product  being  developed  or  the  service  being  provided 

•  planning  and  preparation  necessary  to  execute  a  project  or  process 

•  execution  of  tasks  and  activities 

•  operational  and  business  environments 

•  capacity  and  capability  to  manage  unpredictable  events 
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Appendix  C:  Protocol  Structure  and  Nomenclature 


Information 
FOR  ALL  Phases 


Table  1: 

Information 
FOR  Phase  2 
Activities 


Table  2: 


Table  1  describes  the  information  provided  for  all  phases  of  the  as¬ 
sessment  protocol. 


Information  Type 

Description 

Introduction 

A  brief  introduction  describing  the  key  aspects  of  the  phase 

Objectives 

Key  objectives  for  the  phase  worded  as  questions 

Dataflow 

A  high-level  dataflow  diagram  for  the  protocol  phase 

Note:  For  Phase  2  of  an  assessment  protocol,  a  detailed 
dataflow  of  all  activities  is  also  provided. 

Inputs 

Data  that  are  required  by  a  protocol  phase 

Constraints 

The  limitations  imposed  on  a  protocol  phase  or  activity 

Resources 

Procedures,  plans,  artifacts,  tools,  people,  and  other  re¬ 
sources  that  support  execution  of  a  protocol  phase 

Outputs 

Data  that  are  produced  by  a  protocol  phase 

Key  activities 

A  brief  description  of  the  activities  performed  during  the 
phase 

Information  Types  for  all  Assessment  Phases 


Table  2  describes  the  information  included  for  each  Phase  2  activity. 
The  same  constraints  and  resources  apply  to  all  Phase  2  activities. 
Therefore,  descriptions  of  constraints  and  resources  are  presented  in 
the  information  for  Phase  2,  but  are  not  replicated  in  the  information 
for  each  individual  Phase  2  activity. 


Information  Type 

Description 

Introduction 

A  brief  introduction  describing  the  key  aspects  of  the  proto¬ 
col  activity 

Objectives 

Key  objectives  for  the  protocol  activity  worded  as  questions 

Dataflow 

A  high-level  dataflow  diagram  for  the  protocol  activity 

Inputs 

Data  that  are  required  by  a  protocol  activity 

Outputs 

Data  that  are  produced  by  a  protocol  activity 

Techniques 

A  brief  description  of  the  types  of  techniques  that  can  be 
used  to  conduct  the  protocol  activity 

Information  Types  for  Each  Phase  2  Activity 
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Dataflow 

Structure 


Figure  28: 

Dataflow 

Identifiers 


Table  3: 


Figure  28  depicts  the  data  types  included  in  a  protocol  dataflow.  The 
same  data  types  are  used  when  documenting  the  dataflow  for  a  proto¬ 
col  phase  or  an  activity. 


Constraints 


Inputs 


Outputs 


Protocol  Data  Types 


Each  input,  output,  constraint,  and  resource  listed  in  a  dataflow  is 
represented  by  an  identifier,  which  includes  a  prefix  and  a  number.  The 
prefix  is  based  on  the  type  of  data  and  the  number  represents  a  data 
element.  Table  3  illustrates  the  prefixes  used  in  each  assessment  phase. 


Assessment  Phase 

Prefixes 

Phase  1 

PRI  is  an  input. 

PRO  is  an  output. 

C  is  a  constraint. 

R  is  a  resource. 

Phase  2 

/  is  an  input. 

N  is  an  output  generated  by  one  of  Phase  2's  activities  that 
is  not  a  final  output  of  the  phase.  (It  is  called  an  internal 
output.) 

O  is  a  final  output  of  Phase  2. 

C  is  a  constraint. 

PRO  is  an  output  of  Phase  1  that  either  acts  as  a  constraint 
or  is  used  as  a  resource  in  Phase  2. 

Phase  3 

PAI  is  an  input. 

PAO  is  an  output. 

C  is  a  constraint. 

R  is  a  resource. 

Dataflow  Prefixes 
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Examples  of 

Dataflow 

Identifiers 


Table  4  illustrates  the  convention  for  documenting  dataflow  identifiers 
in  MAAP. 


Dataflow  Identifier 

Description 

PR02  Assessment 

scope 

The  second  output  of  Phase  1 .  It  also  acts  as  a  constraint 
for  all  Phase  2  activities. 

PR06  MAAP  assess¬ 
ment  procedures 

The  sixth  output  of  Phase  1 .  It  also  acts  as  a  resource  for  all 
Phase  2  activities. 

Cl  Assessment  con¬ 
straints 

The  first  constraint  for  the  protocol.  It  can  apply  to  any 
phase  or  activity. 

12  Mission  documenta¬ 
tion 

The  second  input  of  Phase  2.  It  is  also  an  input  to  one  of 
Phase  2’s  activities. 

N2  Data  from  docu¬ 
mentation 

The  second  interim  output  of  Phase  2.  It  is  also  an  input  to 
several  of  Phase  2’s  activities. 

PA01  Communicated 

assessment  results 

The  first  output  of  Phase  3. 

Table  4:  Dataflow  Identifier  Examples 
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